Methods and apparatus to reflect routes from a remotely located virtual route reflector

ABSTRACT

Routes are reflected from a virtual route reflector. For instance, topology information and external route information are requested by a virtual route reflector remote from an autonomous system. The external route information identifies border routers through which a remote destination can be reached. Using the topology information, a first path can be selected from among paths emanating from a selected node in the autonomous system, the paths exiting the autonomous system at respective border routers of the border routers. Further, a route to the remote destination can be advertised from the virtual route reflector to a client router in the autonomous system, the route including a first border router at which the first path exits the autonomous system.

RELATED APPLICATIONS

The subject patent application is a continuation of, and claims priority to each of, U.S. patent application Ser. No. 16/058,720, filed Aug. 8, 2018, and entitled “METHODS AND APPARATUS TO REFLECT ROUTES FROM A REMOTELY LOCATED VIRTUAL ROUTE REFLECTOR,” which is a continuation of U.S. patent application Ser. No. 14/812,426 (now U.S. Pat. No. 10,069,716), filed Jul. 29, 2015, filed Jan. 12, 2018, and entitled “METHODS AND APPARATUS TO REFLECT ROUTES FROM A REMOTELY LOCATED VIRTUAL ROUTE REFLECTOR,” the entireties of which applications are hereby expressly incorporated by reference herein.

TECHNICAL FIELD

This disclosure relates generally to route reflectors, and, more particularly, to methods and apparatus to reflect routes from a remotely located virtual route reflector.

BACKGROUND

“Hot potato” routing is a term used to describe a method by which a route reflector in an autonomous system can select a routing path from among multiple routing paths to a remote destination. The method aims to reduce traffic inside of the autonomous system by transmitting out-bound traffic as quickly as possible. When the route reflector learns that a remote destination can be reached via either a first edge router representing a first point of egress or a second edge router representing a second point of egress, the route reflector selects one of the first or the second edge routers and then notifies a set of client routers that the remote destination can be reached via the selected edge router. Employing hot potato routing, the route reflector selects, and advertises to the client routers, the nearest of the first and second edge routers thereby selecting the nearest point of egress of the autonomous system. As a result of selecting the nearest point of egress, the client routers cause communications intended for the remote destination to exit the autonomous system as quickly as possible.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an example communication system network having a core backbone network, example first, second and third autonomous system networks and an example virtual route reflector residing in a data center.

FIG. 2 is a block diagram of an example implementation of the example virtual route reflector illustrated in FIG. 1.

FIG. 3 is a schematic diagram of a portion of the example communication system network of FIG. 1 in which the example second autonomous system network and the example third autonomous system network are illustrated in greater detail.

FIG. 4 is a flowchart representative of first example computer readable instructions that can be executed by the example virtual route reflector illustrated in FIG. 1, FIG. 2 and/or FIG. 3.

FIG. 5 is a flowchart representative of second example computer readable instructions that can be executed by the example virtual route reflector illustrated in FIG. 1, FIG. 2 and/or FIG. 3.

FIG. 6 is a flowchart representative of third example computer readable instructions that can be executed by the example virtual route reflector illustrated in FIG. 1, FIG. 2 and/or FIG. 3.

FIG. 7 is a flowchart representative of fourth example computer readable instructions that can be executed by the example virtual route reflector illustrated in FIG. 1, FIG. 2 and/or FIG. 3.

FIG. 8 is a block diagram of an example processor platform structured to execute the example machine readable instructions of FIGS. 4, 5 6 and/or 7 to implement the example virtual route reflector of FIGS. 1, 2 and/or 3.

Wherever possible, the same reference numbers will be used throughout the drawing(s) and accompanying written description to refer to the same or like parts.

DETAILED DESCRIPTION

The methods, apparatus and systems disclosed herein provide ways to perform hot potato routing that permits a route reflector to be placed anywhere relative to, and even distant from, a set of client routers served by the route reflector without affecting the efficiency of routing path selection.

Some example methods to virtually reflect routes disclosed herein include requesting, at a route reflector remote from an autonomous system, topology information and external route information from the autonomous system. The external route information identifies a plurality of border routers through which a remote destination can be reached. Example methods also include selecting, using the topology information at the route reflector, a first path from among a plurality of paths emanating from a selected node in the autonomous system. The plurality of paths exit the autonomous system at respective border routers of the plurality of border routers. Some example methods further include advertising, from the route reflector to a client router in the autonomous system, a route to the remote destination. The advertised route includes a first border router at which the first path exits the autonomous system.

In some examples, the first border router is determined to be a nearest point of egress from the autonomous system relative to the selected node. In some examples the topology information is first topology information, the autonomous system is a first autonomous system and the method also includes requesting, at the route reflector, second topology information from a second autonomous system. In some such examples, the first path is determined based on the first topology and the second topology and the remote destination is located in the second autonomous system.

In some further examples, the first topology information is associated with a first interior gateway protocol, the second topology is associated with a second interior gateway protocol, and the first and second interior gateway protocols are different protocols.

In some examples, the first topology information is associated with an interior gateway protocol and requesting the topology information includes initiating a border gateway protocol session with a second border router located on a border of the first autonomous system.

In some examples, selecting a first path includes virtually positioning the route reflector at a location associated with the selected node and determining a cost associated with each of the plurality of paths emanating from the selected node. In some such examples, the first path has the lowest cost.

In some examples, the client router is a first client router, the route is a first route, the selected node is a first node, and the location is a first location. In some such examples, the method further includes virtually positioning the route reflector at the second location at which the second node is located and determining a cost associated with a plurality of paths emanating from the second node and exiting the autonomous system at respective border routers of the plurality of border routers. Some such examples can further include selecting a second path based on the cost determined for the second path and advertising, from the route reflector to a second client router in the autonomous system, a second route to the remote destination. The second route includes a second border router at which the second path exits the autonomous system.

Hot potato routing is a generally effective routing technique when the router reflector is located near its clients. However, the technique can become less effective as the distance between the route reflector and the route reflector's clients increases. For example, a route reflector may determine that between a first edge router and a second edge router that are both able to reach a same remote destination, the first edge router is nearer to itself than the second edge router. As a result, the route reflector advertises the first edge router to the clients of the route reflector. Yet one or more of the route reflector's clients may actually be nearer to the second edge router. When this occurs, some communications to the remote destination will not exit the autonomous system at a nearest point of egress thereby causing the autonomous system to support more traffic than necessary. As a result, network designers looking to utilize hot potato routing attempt to place each route reflector within a desired distance of its clients. For example, each point of presence in an autonomous system having multiple points of presence, is typically equipped with a route reflector. Additionally, large autonomous systems typically have multiple route reflectors strategically placed at various geographical locations in the autonomous system.

Unfortunately, commercially available route reflectors are typically expensive. Thus, it would be desirable to limit the number of route reflectors, yet still be able to achieve effective hot potato routing. The methods systems and apparatus disclosed herein allow the replacement of existing, physical route reflectors with virtualized route reflectors that can be implemented as software installed on any hardware platform capable of operating as a router. Thus, the need to buy expensive, commercially available route reflectors is eliminated.

Moreover, the virtual route reflectors disclosed herein are programmed to serve clients located within a physically remote autonomous system using topology information obtained from the autonomous system. In some examples, the virtual route reflectors are programmed to obtain the topology information from the remote autonomous system, to select a node within the autonomous system based on the topology information, and to operate as though the virtual route reflector were located at the selected node when making routing selections. As a result, the virtual route reflector operates as though it were located within the autonomous system. In some examples, a virtual route reflector disclosed herein causes topology information from a first autonomous system that uses a first interior gateway protocol (IGP) to send first topology information converted into an exterior border gateway protocol (e.g., BGP) and further causes a second autonomous system (contiguous with the first autonomous system) that uses a second IGP to send second topology information converted into BGP, and then uses an accumulated metric associated with the first and second topologies to make best path selections for communications between the first and the second autonomous systems.

FIG. 1 is a schematic diagram illustrating a communication system 100 having a core backbone network (the “core”) 102 coupled to an example first autonomous system (“AS1”) 104, an example second autonomous system, (“AS2”) 106, an example third autonomous system (“AS3”) 108 and an example virtual route reflector (“VRR”) 110 residing in an example data center 112. In some examples, the AS1 104 is coupled to the core 102 via an example first autonomous system boundary router (“ASBR1”) 114, and an example second autonomous boundary router (“ASBR2”) 116 and further coupled to an example first customer edge router (“CE1”) 118 via an example first provider edge router (“PE1”) 120.

In some examples, the AS2 106 is coupled to the core backbone network 102 via an example third autonomous system boundary router (“ASBR3”) 122, and an example fourth autonomous boundary router (“ASBR4”) 124 and is further coupled to the AS3 108 via an example fifth autonomous system boundary router (“ASBR5”) 126, and an example sixth autonomous boundary router (“ASBR6”) 128. In some examples, the AS3 108 is further coupled to an example second customer edge router (“CE2”) 130 via an example second provider edge router (“PE2”) 132.

In some examples, the example AS1 104 includes a set of internal nodes 134 (e.g., an example first internal node (“IN1”) 134A, an example second internal node (“IN2”) 134B, (e.g., an example third internal node (“N3”) 134C, and an example fourth internal node (“IN4”) 134D). In some examples, the internal nodes 134 are fully meshed routers that communicate using an example first interior gateway protocol (“IGP1”). In some examples, the IGP1 is implemented using a protocol referred to as Open Shortest Path First (“OSPF”) version 2 or version 3 and/or is implemented using a protocol referred to as Intermediate System to Intermediate System (IS-IS). The lines of FIG. 1 connecting the ASBR1 114, the ASBR2 116 and the PE1 120 are used to indicate that the ASBR1 114, the ASBR2 116 and the PE1 120 are able to communicate, but do not necessarily indicate that the routers are physically coupled. Likewise, the line connecting the ASBR1 114, the ASBR2 116 and the PE1 120 to the internal nodes 134 of the AS1 104 are intended to indicate that the ASBR1 114 and the ASBR2 116 and the PE1 120 are able to communication with the internal nodes 134 of the AS1 104, but do not necessarily indicate that the routes are physically coupled.

In some examples, the example AS2 106 also includes a set of internal nodes 136 (represented collectively using an ellipse in FIG. 1 and represented individually in FIG. 3 as described hereinbelow) that communicate using an example second interior gateway protocol (“IGP2”) which may be implemented using, for example, OSPF v2/v3, IS-IS etc. Likewise, the example AS3 108 includes a set of internal nodes 138 (represented collectively via an ellipse in FIG. 1 and individually in FIG. 3 as described hereinbelow) that communicate using a third interior gateway protocol (“IGP3”) which may be implemented using, for example, OSPF v2/v3, IS-IS etc.

In some examples, the example autonomous system boundary routers (e.g., the example ASBR1 114, and the example ASBR2 116, the example ASBR3 122, the example ASBR4 124, the example ASBR5 126, and the example ASBR6 128) and the example provider edge routers (e.g., the example PE1 120 and the example PE2 132) use an exterior border gateway protocol (“EBGP”) to learn routes to destinations located outside of the respective example autonomous systems (e.g., the example AS1 104, the example AS2 106, the example AS3 108, etc.). Thus, the ASBR1 114, the ASBR2 116, the ASBR3 122, the ASBR4 124, the ASBR5 126, the ASBR6 128, the PE1 120 and the PE2 132 provide a gateway by which routers within the respective autonomous systems (e.g., the example AS1 104, the example AS2 106 and the example AS3 108) can reach exterior destinations (i.e., destinations outside of AS1 104, the AS2 106, and the AS3 108, respectively). Additionally, the border routers (e.g., the ASBR1 114, the ASBR2 116, the ASBR3 122, the ASBR4 124, the ASBR5 126, and the ASBR6 128, the PE1 120 and the PE2 132) use respective interior gateway protocols (e.g., IGP1, IGP2, IGP3) to communicate with internal nodes of the respective autonomous systems (e.g., AS1 104, the AS2 106 and the AS3 108). Thus, for example, the ASBR1 114, the ASBR2 116 and the PE1 120 communicate with the internal nodes (e.g., the example IN1 134A, the example IN2 134B, the example IN3 134C, and the example IN4 134D) of the AS1 104 using the IGP1. Likewise, the ASBR3 122, the ASBR4 124, the ASBR5 126 and the ASBR6 128 communicate with the internal nodes 136 (see FIG. 3) of the AS2 106 using the IGP2, and the ASBR5 126, the ASBR6 128 and the PE2 132 communicate with the internal nodes 138 of the AS3 108 using the IGP3.

In some examples, the example virtual router reflector 110 initiates a BGP communication session with the example autonomous system boundary router, ASBR1 114. During the communication session, the virtual router reflector 110 requests topology information for AS1 104. Responsive to the request, the ASBR1 114 redistributes the topology information for AS1 104 into a format that is transferrable using an EBGP. Redistribution, as used herein, refers to the process by which the internal topology of an autonomous system is converted into a protocol for suitable transmission to an external destination. One such example protocol is BGP-LS. A method used to redistribute topology information from an autonomous system to a format suitable for transmission via BGP is described in the Internet Draft distributed by the Internet Engineering Task Force (IETF) titled, “North-Bound Distribution of Link-State and TE Information using BGP, draft-ietf-idr-ls-distribution-10.” Although BGP-LS is used as an example protocol for transmitting the topology information of the AS1 104 to the virtual router reflector 110, any routing communication protocol capable of permitting the transmission of autonomous system topology information to external network(s) may be used.

In addition to requesting the first topology information, the virtual route reflector 110 requests external routing information from the ASBR1 114. In some examples, the ASBR1 114 responds to the request by delivering a set of routes to external network destinations (i.e., network destinations that are external to the autonomous system AS1 104). In some examples, the set of routes delivered by the ASBR1 114 include a list of external network destinations that can be reached by any of the border routers of the first autonomous system (e.g., the ASBR1 114, the ASBR2 1165, the PE1 120) and further identifies the respective border routers that can reach each such external network destination.

The example virtual router reflector 110 selects a node within the example autonomous system, AS1 104, and uses the location of that node within the topology of the AS1 104 as a virtual position (also referred to as a pseudo location). Thus, the virtual route reflector 110 “pretends” to be located at the selected node when determining a set of paths to be used to reach external network destinations that are accessible via the example ASBR1 114, the example ASBR2 116 and/or the example PE1 120. In some such examples, the virtual route reflector 110 uses the list of external routes to select a target network destination from the list of external network destinations and further uses the list of routes to identify the border routers (e.g., the ASBR1 114, the ASBR2 116, and the PE2 120) of AS1 104 that are capable of reaching the target network destination.

Next, the example virtual route reflector 110 uses any desired method including, for example, Dijkstra's algorithm to select/determine a “best” path from the selected node (the virtual position) to one of the border routers (e.g., the ASBR1 114, the ASBR2 116, and the PE2 120) through which the target network destination can be reached. In some such examples, the best path is selected as the path from the selected node (at which the virtual route reflector is virtually positioned) to the nearest of the autonomous system border routers (e.g., the ASBR1 114, the ASBR2 116, and the PE2 120) that are capable of “reaching” the desired exterior destination to thereby achieve hot potato routing. The virtual route reflector 110 then transmits, via the core backbone 102, the selected best path to the ASBR1 114, for example, for distribution to the internal nodes (e.g., IN1 134A, the IN2 134B, the IN3 134C, and the IN4 134D) of the AS1 104 for use in reaching the desired destination.

In some examples, virtual route reflector selects, from the list of external routes, the ASBR3 122 associated with the AS2 106 as the target destination and further uses the list of external routes to determine that either of the ASBR1 114 or the ASBR2 116 can be used by the internal nodes 134 of the AS1 104 to reach the target destination (e.g., both the ASBR1 114 and the ASBR2 116 advertise a route(s) to the target network destination). In some such examples, the virtual route reflector 110 selects the first node IN1 134A as the node from which to calculate a best path. In some such examples, the virtual route reflector 110 uses a path selection algorithm to determine whether a first link (“link1”) between the first node IN1 134A and the ASBR1 114 is shorter than a second link (“link2”) between the first node IN1 134A and the ASBR2 116. In some such examples, the virtual route reflector 110 determines that the link1 is the shorter path and thus the link1 is selected as the best path. In some examples, the path selector builds a path tree and sets itself as the origin of the tree to identify the shortest path. Thus, the ASBR1 114 associated with the link1 represents the “nearest” point of egress from the virtual position (e.g., the first node IN1 134A). In some such examples, the virtual route reflector 110 selects a best route to the target network destination as being the route that travels through the ASBR1 114 and subsequently advertises that route to the internal nodes 134 of the AS1 104. The internal nodes 134 of the AS1 104 then use that route for transmission of packets intended for the target network destination, ASBR3 122.

In some examples, instead of using a single selected node as the virtual position of the virtual route reflector 110, the virtual route reflector 110 iteratively performs the path selection process. During each such iteration, the virtual route reflector 110 virtually positions itself at one of the internal nodes 134 and subsequently selects a best path extending from the virtual position to the target network destination. The process is repeated for each of the internal nodes 134 until a best path is selected for each of the internal nodes 134. For example, the virtual route reflector 110 may determine that although the link1 is the best path by which the first node IN1 134A can reach the target network destination, a link3 represents a best path by which IN2 134B can reach the target network destination. Consequently, the virtual route reflector 110 advertises, to the IN2 134B, a route that extends through the ASBR2 134B to reach the target network destination. In this manner, the virtual route reflector can determine a best path for each of the individual internal nodes 134 to reach each external network destination that is advertised by the border routers (e.g., the ASBR1 114, the ASBR2 116, the PE1 120) of the AS1 104.

As described further below, in some examples, the example virtual route reflector 110 performs route reflection operations for multiple autonomous systems. In some such examples, the virtual route reflector 110 obtains network topology information from multiple autonomous systems (e.g., the example AS1 104, the example AS2 106 and the example AS3 108). In some such examples, the virtual route reflector 110 uses the topology information of each autonomous system (e.g., the AS1 104, the AS2 106, and the AS3 108) to calculate paths to be used by the internal nodes 134, 136, 138 of each of the multiple autonomous systems (e.g., AS1 104, the AS2 106, and the AS3 108) to reach target destinations exterior to the autonomous systems (e.g., AS1 104, the AS2 106, and the AS3 108).

As described in greater detail below with reference to FIG. 3, in some examples, the example virtual route reflector 110 uses second topology information of the example AS2 106 and third topology information of the AS3 108 to select a “best” path from a selected one of the internal nodes 136 of the AS2 106 to a selected one of the internal nodes 138 of the AS3 108. In some such examples, the virtual route reflector identifies the autonomous system boundary router (e.g., ASBR5 126) through which the path travels and subsequently advertises that router to the selected ones of the internal nodes of the AS2 106 and the AS3 108 for use in communicating therebetween.

A block diagram illustrating an example implementation of the example virtual route reflector 110 of FIG. 1 is shown in FIG. 2. In some examples, the virtual route reflector 110 includes an example network interface 202, an example topology and route collector 204, an example topology database storage 206, an example external routes database storage 208, an example virtual positioner 210, an example path selector 212, an example path storage 214, and an example route advertiser 216 coupled together via a communication bus 218.

Referring now to FIG. 1 and FIG. 2, in some examples, the virtual route reflector 110 operates as a route reflector for the first autonomous system, AS1 104. In some such examples, the example network interface 202 of the virtual route reflector 110 begins a BGP-LS communication session with any of the boundary routers of the AS1 104 (e.g., any of the ASBR1 114, the ASBR2 116 and the PE1 120). In some such examples, the network interface 202 begins the session with the ASBR1 114. During the communication session, the example topology collector 204 requests that the ASBR1 114 transmit topology information describing the topology of the AS1 104 (“the AS1 topology information”). In response, the ASBR1 114 transmits the AS1 topology information in any exterior gateway protocol capable of carrying autonomous system topology information such as, for example, BGP-LS. In some such examples, the AS1 topology information identifies the example nodes of the AS1 104 (e.g., the IN1 134A, the IN2 134B, the IN3 134C, the IN4 134D) and further identifies links by which the IN1 134A, the IN2 134B, the IN3 134C, the IN4 134D are coupled. In some examples, the AS1 topology information also identifies a cost (also called a metric), associated with each link. The cost of associated with a link represents the overhead required to send packets across that link. Typically, a higher cost is associated with a lower bandwidth and a lower cost is associated with a higher bandwidth. In some such examples, the cost information can be transmitted using an accumulated internal gateway protocol (AIGP) attribute which can be set by enabling the ASBR1 to process AIGP information. The topology and route collector 204 causes the topology information to be stored in the example topology database 206.

In some examples, during the communication session with the example ASBR1 114, the example topology and route collector 204 also requests that the ASBR1 114 transmit external network routing information identifying external routes that are advertised by the border routers of the first autonomous system AS1 104 (e.g., the ASBR1 114, the ASBR2 116, the PE1 120). Thus, for example, the external network routing information identifies external network destinations and each of the border routers of the first autonomous system AS1 104 (e.g., the ASBR1 114, the ASBR2 116, the PE1 120) that are capable of “reaching” the external network destinations. The external topology and route collector 204 stores the external routes in the example external routes database 208.

In some such examples, the example virtual positioner 210 of the virtual route reflector 110 selects any node (e.g., IN1 134A) in the AS1 104 and thereafter the virtual route reflector 110 uses the location of that node (IN1 134A) as a starting location in determining a nearest point of egress from the AS1 104 to the core backbone 102, for example. By using the location of the node IN1 134A as the starting location in determining a nearest point of egress from the AS1 104, the virtual route reflector is essentially “pretending” to be located at the first node IN1 134A. As used herein, when the virtual route reflector 110 “pretends” to be located at the first node IN1 134A, the virtual route reflector 110 is “virtually positioning” itself at the first node IN1 134A. Thus, the location at which the virtual route reflector 110 is “pretending” to be is also referred to as the “virtual position” of the virtual route reflector 110.

In some examples, the example path selector 212 of the virtual route reflector 110 uses the example external routes database 208 to identify an external network destination that can be reached by one or more of the border routers of the example AS1 104 (e.g., the example ASBR1 114, the example ASBR2 116, the example PE1 120). In some examples, the external network destination, also referred to as the target network destination, is the example ASBR3 122 associated with the AS2 106. In some such examples, the example path selector determines that the target network destination, ASBR3 122, can be reached via either the ASBR1 114 or the ASBR2 116. Next, the path selector 212 uses the node, link and cost information stored in the example topology database 210 to identify a “best” path from among the example link1 that extends between the virtual position (e.g., the location of the first node IN1 134A) and the ASBR1 114, and the example link2 that extends between the virtual position (e.g., the location of the first node IN1 134A) and the ASBR2 116.

In some examples, the example path selector 212 uses any technique, such as, for example, Dijkstra's algorithm, to determine the “best” path. In some such examples, the “best” path is identified as the path having the lowest associated cost. The path selector 212 causes information identifying the “best” path to be stored in the example path storage 214 of the virtual route reflector 110. Information identifying the “best” path can include the target destination and the ASBR associated with the “best” path. Thus, for example, if the link1 is determined to be the “best” path (as opposed to the linke2), then the information identifying the “best” path will include information identifying the address of the target network destination (e.g., the ASBR3 122) and information identifying the address of the boundary router associated with the link1 (in this example, the address of the ASBR1 114). The example route advertiser 216 incorporates the information identifying the best path stored in the path storage into an appropriate route protocol for transmission to the ASBR1 114. The example route advertiser 216 then advertises the generated route, via the example network interface 202, to the ASBR1 114, the ASBR2 116 and/or the PE1 120 for distribution to the internal nodes (e.g., the IN1 134A, the IN2 134B, the IN3 134C and the IN4 134D) of the AS1 104. Subsequently, the internal nodes (e.g., the IN1 134A, the IN2 134B, the IN3 134C and the IN4 134D, etc.) of the AS1 104 use the advertised route to transmit messages to the target destination, ASBR3 122. Thus, the boundary router (in this example, ASBR1 114) nearest to the virtual location will be used as the point of egress for messages transmitted to the ASBR3 122 by the nodes 134, to thereby effect hot potato routing.

In some examples, the example virtual route reflector 110, instead of virtually positioning itself at a single one of the internal nodes 134 of the AS1 104, virtually positions itself at each of the internal nodes 134 of the AS1 104 in an iterative fashion and determines which of the ASBR1 114 and the ASBR2 116 are nearest to each such internal node (e.g., the IN1 134A, the IN2 134B, the IN3 134C, or the IN4 134D) of the AS1 104. Based on that information, the virtual route reflector 110 advertises, to each respective internal node, a respective route by which the target network destination can be reached. In some such examples, a first route to reach the target destination that is advertised by the virtual route reflector 110 to the IN1 134A includes the boundary router (either the ASBR1 114 or the ASBR2 116) that is closest to the internal node IN1 134A. Likewise, a second route to reach the remote destination that is advertised by the virtual route reflector 110 to the IN2 134B includes the boundary router (either the ASBR1 114 or the ASBR2 116) that is closest to the internal node IN2 134B. Additionally, a third route and a fourth route to reach the remote destination advertised to the IN3 134C and the IN4 134D, respectively, includes the boundary router (either the ASBR1 114 or the ASBR2 116) that is closest to the internal node IN3 134B and the internal node IN4 134D, respectively.

In some such examples, the example virtual positioner 206 of the virtual route reflector 110 virtually positions itself at the location of the first internal node IN1 134A. The shortest path selector then uses the topology information stored in the example matrix storage 210 to determine which of the boundary routers (the ASBR1 134A and the ASBR2 134B) are nearest to the first internal node IN1 134A (e.g., to select the shortest path from the IN1 134A to a point of egress (boundary) router from the AS1 104 that is capable of reaching the remote destination. The path to the nearest of the boundary routers is selected as the shortest path and stored in the example path storage 214. The example route advertiser incorporates the shortest path into the route to be advertised, via the network interface 202, to the internal node IN1 134A. To identify the shortest path from each of the remaining internal nodes to a boundary router (e.g., the ASBR1 134A or the ASBR2 134B), the operations are repeated for each internal node (e.g., the virtual route reflector 110 virtually positions itself at the location of each internal node of the AS1 104), determines whether the ASBR1 134A or the ASBR2 134B is closer to the virtual position (e.g., determines which of a first path from the virtual position to the ASBR1 134A and a second path from the virtual position to the ASBR2 134B is shortest), incorporates the shortest path into the route, and advertises, to the internal node, the route by which the remote destination can be reached.

The example second autonomous system 106 and the example third autonomous system AS3 108 of FIG. 1 are illustrated in further detail in FIG. 3. In some examples, the example internal nodes 136 in the second autonomous system 106 include an example fifth internal node IN5 136A, an example sixth internal node IN6 136B, an example seventh internal node IN7 136C, and an example eighth internal node IN8 136D. The example internal nodes 138 in the third autonomous system 106 include an example ninth internal node IN9 138A, an example tenth internal node IN10 138B, an example eleventh internal node IN11 138C, an example twelfth internal node IN12 138D and an example thirteenth internal node IN13 138E.

Referring now to FIG. 2 and FIG. 3, in some examples, the example virtual route reflector 110 residing in the example data center 112 operates as a route reflector for the example second autonomous system AS2 106 and for the example third autonomous system 108. In some such examples, when identifying a route from any node (e.g., the fifth node IN5 136A) in the second autonomous system AS2 106 to any other node (e.g., the eleventh internal node IN11 138C located in the third autonomous system AS3 108, the virtual route reflector 110 uses second topology information collected from the second autonomous system AS2 106 and uses third topology information collected from the third autonomous system AS3 108 to determine a shortest path between the fifth node IN5 136A and the eleventh node 138C. In some such examples, the virtual route reflector 110 selects the boundary router located on the shortest path (e.g., either the ASBR5 126 or the ASBR6 128) as the point of egress from the second autonomous system AS2 106 to be used by the fifth node IN5 136A when transmitting messages to the eleventh node IN11 138C. In some examples, the ASBR5 126 is located on the shortest path between the fifth node IN5 136A and the eleventh node IN11 138C. In some such examples, the virtual route reflector 110 generates and advertises a route to the fifth node IN5 136A that identifies the ASBR5 126 as the boundary router to which the fifth node IN5 136A is to deliver messages when the intended remote destination for the messages is the eleventh node IN11 138C located in the second autonomous system AS2 106. Similarly, the virtual route reflector 110 generates and advertises a route to the eleventh node IN11 138C that identifies the ASBR5 126 as the boundary router to which the eleventh node IN11 138C is to deliver messages when the intended remote destination for the messages is the fifth node IN5 136A located in the third autonomous system AS3 108.

Referring now to FIG. 1, FIG. 2 and FIG. 3, in some example, the example virtual route reflector is configured to operate as a route reflector for the example first autonomous system AS1 104, the example second autonomous system AS2 106, and the example third autonomous system AS3 108. In some such examples, the example topology and route collector 204 of the virtual route reflector collects first topology information, second topology information and third topology information from any of the boundary routers associated with the first autonomous system AS1 104, the second autonomous system AS2 106, and the third autonomous system AS3 108, respectively. Additionally, the example topology and route collector 204 collects external routing information identifying external network destinations reachable by one or more of the boundary routers associated with the AS1 104, the AS2 106 and the AS3 108, respectively, and further identifying the respective boundary routers through which each respective, external network destination can be reached. The topology and route collector 204 stores the topology information in the example topology database 206 and stores the external routing information in the example external routes database 208.

Additionally, the example virtual positioner 210 virtually positions itself in each of the three autonomous systems (AS1 104, AS2 106, AS3 108) in the manner described above. Using the virtual positions, the topology information and the external routing information, the example path selector 212 determines best paths from one or more of the nodes (e.g., IN1 134A, IN2 134B, IN3 134C, IN4 134D) in the first autonomous system AS1 104 to one or more of the nodes (e.g., IN5 136A, IN6 136B, IN7 136C, IN8 136D) in the second autonomous system AS2 106 and to one or more of the nodes in the third autonomous system AS3 108. Likewise, the path selector 212 identifies a set of shortest paths from one or more of the nodes (e.g., IN5 136A, IN6 136B, IN7 136C, IN8 136D) in the second autonomous system AS2 106 to one or more of the nodes (e.g., IN1 134A, IN2 134B, IN3 134C, IN4 134D) in the first autonomous system AS1 104 and to one or more of the nodes (e.g., IN9 138A, IN10 138B, IN11 138C, IN12 138D, IN13 138E) in the third autonomous system AS3 108. The shortest paths are stored in the example path storage 214 and then incorporated into a set of routes by the example route advertiser 216 for transmission to respective ones of the autonomous systems (e.g., the AS1 104, the AS2 104, the AS3 106).

While an example manner of implementing the virtual route reflector 110 of FIG. 1 and FIG. 3 is illustrated in FIG. 2, one or more of the elements, processes and/or devices illustrated in FIG. 2 may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the example network interface 202, the example topology and route collector 204, the example topology database storage 206, the example external routes database storage 208, the example virtual positioner 210, the example path selector 212, the example path storage 214 and the example route advertiser 216 and/or, more generally, the example virtual router reflector 110 of FIG. 2 may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example network interface 202, the example topology and route collector 204, the example topology database storage 206, the example external routes database storage 208, the example virtual positioner 210, the example path selector 212, the example path storage 214 and the example route advertiser 216 and/or, more generally, the example virtual route reflector 110 could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example network interface 202, the example topology and route collector 204, the example topology database storage 206, the example external routes database storage 208, the example virtual positioner 210, the example path selector 212, the example path storage 214, and/or the and the example route advertiser 216 is/are hereby expressly defined to include a tangible computer readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc. storing the software and/or firmware. Further still, the example virtual route reflector 110 of FIG. 1 and FIG. 3 may include one or more elements, processes and/or devices in addition to, or instead of, those illustrated in FIG. 3, and/or may include more than one of any or all of the illustrated elements, processes and devices.

Flowcharts representative of example machine readable instructions for implementing the virtual route reflector 110 of FIGS. 1, 2 and 3 are shown in FIGS. 4, 5, 6 and 7. In these examples, the machine readable instructions comprise a program for execution by a processor such as the processor 1012 shown in the example processor platform 1000 discussed below in connection with FIG. 8. The program may be embodied in software stored on a tangible computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), a Blu-ray disk, or a memory associated with the processor 1012, but the entire program and/or parts thereof could alternatively be executed by a device other than the processor 1012 and/or embodied in firmware or dedicated hardware. Further, although the example programs are described with reference to the flowcharts illustrated in FIGS. 4, 5, 6 and 7 many other methods of implementing the example virtual route reflector 110 may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined.

As mentioned above, the example processes of FIGS. 4, 5, 6 and 7 may be implemented using coded instructions (e.g., computer and/or machine readable instructions) stored on a tangible computer readable storage medium such as a hard disk drive, a flash memory, a read-only memory (ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, a random-access memory (RAM) and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term tangible computer readable storage medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media. As used herein, “tangible computer readable storage medium” and “tangible machine readable storage medium” are used interchangeably. Additionally or alternatively, the example processes of FIGS. 4, 5, 6, and 7 may be implemented using coded instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media. As used herein, when the phrase “at least” is used as the transition term in a preamble of a claim, it is open-ended in the same manner as the term “comprising” is open ended.

The program 400 of FIG. 4 represents a method by which the example virtual route reflector 110 performs route reflection for an autonomous system (e.g., the AS1 104) from a location outside of the AS1 104 by virtually positioning itself at a single node located within the AS1 104. With reference also to FIG. 1 and FIG. 2, the method begins at a block 402 after which the example network interface 202 (see FIG. 2) of the example virtual route reflector 110 (see FIG. 2) initiates a BGP communication session with the example autonomous system boundary router, ASBR1 114 (see FIG. 1) (block 404). During the communication session, the example topology collector 204 requests topology information for the AS1 104 (block 406). Responsive to the request, the ASBR1 114 accesses one or more topology databases (e.g., a link state database, a traffic engineering database, etc.) to obtain first topology information describing the topology of the first autonomous system AS1 104. In addition, the ASBR1 114 redistributes the first topology information into a format that is transferrable using an EB GP such as, for example BGP-LS. BGP-LS is a protocol into which topology information of an autonomous system can be formatted for transmission outside of the autonomous system. A method used to redistribute topology information from an autonomous system to a format suitable for transmission via BGP is described in the Internet Draft distributed by the Internet Engineering Task Force (IETF) titled, “North-Bound Distribution of Link-State and TE Information using BGP, draft-ietf-idr-ls-distribution-10,” Although BGP-LS is used as an example protocol for transmitting the topology information of the AS1 104 to the virtual router reflector 110, any routing communication protocol capable of permitting the transmission of autonomous system topology information to external network(s) may be used.

The example topology and route collector 204 then stores the first topology information in the example topology database storage 206 (block 408). In some examples, the topology and route collector 204 generates the topology database by using the first topology information to identify each of the nodes residing in the first AS1 104 (e.g., IN1 134A, IN2 134B, IN3 134C, IN4 134D, etc.) and the links by which the nodes are linked. The topology and route collector 204 also uses the topology information to identify a cost (or metric) associated with each link.

During the communication session with the ASBR1 114, the example topology and route collector 204 also requests that the ASBR1 114 transmit external network routing information identifying external routes that are advertised by the border routers (e.g., the ASBR1 114, the ASBR2 116, the PE1 120) of the first autonomous system AS1 104 (also block 404). Thus, for example, the external network routing information identifies external network destinations and a set of corresponding border routers (e.g., the ASBR1 114, the ASBR2 116, the PE1 120) of the first autonomous system AS1 104 that are capable of “reaching” the external network destinations. The topology and route collector 204 stores the external routes in the example external route database (also block 406).

In some examples, the example virtual positioner 210 of the virtual route reflector 110 uses the first topology information to select a next (or a first, during the first iteration of the program 400) node residing within the example first autonomous system AS1 104 (block 410). The location of the selected node within the AS1 104 will be used as the virtual position of the virtual route reflector 110 as described below. The virtual positioner 206 may select the node at random or using any desired criteria such as, for example, based on a user input, based on a set of rules, etc. As described in greater detail below, the example path selector 212 then uses the location of that node within the topology of the first autonomous system AS1 104 as a virtual position for the virtual route reflector 110 (i.e., the path selector 212 uses the location of the selected node as the location of the virtual route reflector 110) from which to select/calculate a “best” path by which any of the internal nodes (e.g., any of the IN1 134A, the IN2 134B, the IN3 134C and the IN4 134D) of the first autonomous system AS1 104 may reach a target network destination external to the first autonomous system AS1 104.

Additionally, the example path selector 212 uses the external route database stored in the example external route database storage 208 to identify a target network destination, such as the ASBR3 122, that is external to the first autonomous system AS1 104 and that is reachable by one or more of the border routers, such as the ASBR1 114, and the ASBR2 116, of the first autonomous system 104 (block 412). The path selector 212 uses any desired method including, for example, Dijkstra's algorithm to select a “best” path from the virtual position to either of the ASBR1 114 or the ASBR2 116. In some such examples, the best path is selected as the path from the virtual position to the “nearest” of the ASBR1 114 and the ASBR2 116 to thereby achieve hot potato routing. In some such examples, the costs associated with the example link1 between the virtual position and the first ASBR1 114 and the costs associated with the example link2 between the virtual position and the second ASBR2 116 are compared. In some such examples, a lower cost is associated with a shorter distance. Thus, if the cost of the first link is less than the cost of the second link, then the first ASBR1 is determined to be “nearer” to the virtual position and the first link is selected as the “best” path. As a result, the path selector 212 stores information identifying the first link in the example path storage 214 (block 414). In some examples, the information identifying the first link (i.e., the selected path) includes the address of the target network destination (in this example ASBR3 122) and also identifies the address of the border router of AS1 104 that is “nearest” to the virtual position (in this example, ASBR1 114) and any other desired route information. If needed, the example route advertiser 216 then converts the information identifying the selected path into a route protocol or format that is suitable for transmission to the ASBR1 114 (e.g., BGP-LS) (block 416).

Next, the example path selector 212 determines whether there are any external network destinations in the external routes database for which a best path has not yet been selected (block 418). If so, control returns to the block 412 at which the path selector 212 selects a next external network destination from the external routes database storage 208 to be the target network destination and control proceeds thereafter in the manner described above. If a best path has been selected for every external network destination in the external routes database (as determined at the block 418), the example route advertiser 216 provides the route information containing the selected paths to the example network interface 202 for transmission to the ASBR1 114 via the core backbone 102 (block 420) and the method ends (block 422). Upon receipt of the advertised routes, the ASBR1 114 supplies the routes to the internal nodes 134 (e.g., IN1 134A, the IN2 134B, the IN3 134C, the IN4 134D, etc.) of the AS1 104 for use in reaching the corresponding target network destinations. For example, the internal nodes 134 of the AS1 104 will transmit messages intended for the target network destination of the ASBR3 122 to the ASBR1 114 for subsequent transmission to the ASBR3 122 based on the “best” path selected for the ASBR3 122. Although “best path” as used herein typically refers to a path having a lower cost than other paths, the terms could instead be used to describe a path meeting any desired criteria.

Referring now to FIG. 5, a program 500 represents a method by which the example virtual route reflector 110 performs route reflection for an autonomous system (e.g., the AS1 104) from a location outside of the AS1 104 by virtually positioning itself at multiple locations within the AS1 104. Referring also to FIG. 1 and FIG. 2, the method begins at a block 502 after which the example network interface 202 (see FIG. 2) of the example virtual route reflector 110 (see FIG. 1 and FIG. 2) initiates a BGP communication session with the example autonomous system boundary router, ASBR1 114 (or any of the other border routers of the AS1 104) (see FIG. 1) (block 504). During the communication session, the example topology collector 204 requests first topology information for the AS1 104 (block 506). Responsive to the request, the ASBR1 114 accesses one or more topology databases (e.g., a link state database, a traffic engineering database, etc.) to obtain first topology information describing the topology of the first autonomous system AS1 104. In addition, the ASBR1 114 redistributes the first topology information into a format that is transferrable using an EBGP such as, for example BGP-LS. The example topology and route collector 204 then stores the first topology information in the example topology database storage 206 (block 508). In some examples, the topology and route collector 204 generates the topology database by using the first topology information to identify each of the nodes residing in the first AS1 104 (e.g., IN1 134A, IN2 134B, IN3 134C, IN4 134D, etc.) and the links by which the nodes are linked. The topology and route collector 204 also uses the topology information to identify a cost (or metric) associated with each link.

During the communication session with the ASBR1 114, the example topology and route collector 204 also requests that the ASBR1 114 transmit external network routing information identifying external routes that are advertised by the border routers (e.g., the ASBR1 114, the ASBR2 116, the PE1 120) of the first autonomous system AS1 104 (also block 506). Thus, for example, the external network routing information identifies external network destinations and a set of corresponding border routers (e.g., the ASBR1 114, the ASBR2 116, the PE1 120) of the first autonomous system AS1 104 that are capable of “reaching” the external network destinations. The topology and route collector 204 stores the external routes in the example external route database (also block 508).

In some examples, the example virtual positioner 210 of the example virtual route reflector 110 uses the first topology information to select a next internal node residing within the example first autonomous system AS1 104 (block 510). On the first iteration of the method of FIG. 5, the virtual positioner 210 selects a first of the internal nodes 134 (e.g., IN1 134A)). The location of the selected node within the AS1 104 will be used as the virtual position of the virtual route reflector 110 to select paths as described in greater detail below.

Additionally, the example path selector 212 uses the external route database stored in the example external route database storage 208 to identify a next target network destination (or a first target network destination during the first iteration of the program 500), such as the ASBR3 122, that is external to the first autonomous system AS1 104 and that is reachable by one or more of the border routers, such as the ASBR1 114, and the ASBR2 116, of the first autonomous system 104 (block 512). Next, the path selector 212 uses any desired method including, for example, Dijkstra's algorithm to select a “best” path from the virtual position to either of the ASBR1 114 or the ASBR2 116. In some such examples, the best path is selected as the path from the virtual position to the “nearest” of the ASBR1 114 and the ASBR2 116 to thereby achieve hot potato routing. In some such examples, the costs associated with the example link1 between the virtual position and the first ASBR1 114 and the costs associated with the example link2 between the virtual position and the second ASBR2 116 are compared. In some such examples, a lower cost is associated with a shorter distance. Thus, if the cost of the first link is less than the cost of the second link, then the first ASBR1 is determined to be “nearer” to the virtual position and the first link is selected as the “best” path. As a result, the path selector 212 stores information identifying the first link in the example path storage 214 (block 514). In some examples, the information identifying the first link (i.e., the selected path) includes the address of the target network destination (in this example ASBR3 122) and also identifies the address of the border router of AS1 104 that is “nearest” to the virtual position (in this example, ASBR1 114) and any other desired route information. If needed, the example route advertiser 216 then converts the information identifying the selected path into a route using a routing protocol or format that is suitable for transmission to the ASBR1 114 (e.g., BGP-LS) (block 516).

Next, the example path selector 212 determines whether there are any external network destinations in the external routes database for which a best path has not yet been selected (block 518). If so, control returns to the block 510 at which the path selector 212 selects a next external network destination from the external routes database storage 208 to be the target network destination and control proceeds thereafter in the manner described above. If a best path has been selected for every external network destination in the external routes database (as determined at the block 518), the virtual positioner 210 determines if there are any internal nodes 134 within the autonomous system (e.g., AS1 104) for which best paths have not yet been selected (block 520). If best paths have not yet been selected for any of the internal nodes 134, control returns to the block 510 at which the virtual positioner 210 selects a next internal node (e.g., any of the IN2 134B, the IN3 124C and the IN4 134D that have not yet been processed) and thereafter control proceeds to the blocks subsequent thereto as described above. In this manner, the method represented by the program 500 determines a respective set of best paths by which each of the respective internal nodes of the AS1 104 can reach external network destinations. If, at the block 518, the virtual positioner 210 determines that best paths have been selected for all of the internal nodes 134, the example route advertiser 216 provides the route information containing the selected paths to the example network interface 202 for transmission to the ASBR1 114 via the core backbone 102 (block 522) and the method ends (block 524).

Upon receipt of the advertised routes, the ASBR1 114 supplies the respective routes to the respective internal nodes 134 (e.g., IN1 134A, the IN2 134B, the IN3 134C, the IN4 134D, etc.) of the AS1 104 for use in reaching the corresponding target network destinations. Thus, each of the respective internal nodes 134 is supplied a respective set of best paths for use in reaching a respective, nearest point of egress for each external network destination.

Referring now to FIG. 6, a program 600 represents a method by which the example virtual route reflector 110 performs route reflection for an autonomous system (e.g., the AS2 106) from a location outside of the AS2 106 by virtually positioning itself a location within the AS2 106. As described below, in the method represented by the program 600, the virtual route reflector 110 uses second topology information describing the topology of the AS2 106 and third topology information describing the topology of another autonomous system (e.g., the AS3 108) to determine best paths between nodes located in the AS2 104 and the AS3 106. Referring also to FIG. 1 and FIG. 2, the method 600 begins at a block 602 after which the example network interface 202 (see FIG. 2) of the example virtual route reflector 110 (see FIG. 1 and FIG. 2) initiates a first BGP communication session with the example autonomous system boundary router, ASBR3 122 (or any of the other border routers of the AS2 106) (see FIG. 1) and further initiates a second BGP communication session with the example autonomous system boundary router ASBR5 126 (see FIG. 1) (block 604) of the AS3 108. During the first BGP communication session, the topology and route collector 204 (see FIG. 2) requests second topology information for the AS2 106 (block 606) from the ASBR 3 122 and during the second BGP communication session, the topology and route collector 204 requests third topology information for the AS3 108 from the ASBR5 126 (block 606). In some examples, the virtual route reflector 110 is unable to directly communicate with the ASBR5 126. In some such examples, the virtual route reflector 110 instructs the ASBR3 122 to request the third topology information from the ASBR5 126.

Responsive to the request, the ASBR3 122 accesses one or more topology databases associated with the AS2 106 (e.g., a link state database, a traffic engineering database, etc.) to obtain second topology information describing the topology of the second autonomous system AS2 106. In addition, the ASBR3 122 redistributes the second topology information into a format that is transferrable using an EBGP such as, for example BGP-LS. Similarly, the ASBR5 126 accesses one or more topology databases associated with the AS3 108 (e.g., a link state database, a traffic engineering database, etc.) to obtain the third topology information describing the topology of the third autonomous system AS3 108. In addition, the ASBR5 126 redistributes the third topology information into a format that is transferrable using an EBGP such as, for example BGP-LS.

In some examples, the topology and route collector 204 uses the second topology information to identify each of the nodes residing in the AS2 106 (e.g., IN5 136A, IN6 136B, IN7 136C, IN7 136D, etc.) and the links by which the nodes are coupled. The topology and route collector 204 also uses the second topology information to identify a cost (or metric) associated with each link in the AS2 106. Likewise, the topology and route collector 204 uses the third topology information to identify each of the nodes residing in the AS3 108 (e.g., IN9 138A, IN10 138B, IN11 138C, IN12 138D, IN13 138D etc.) and the links by which the nodes are coupled. The topology and route collector 204 also uses the third topology information to identify a cost (or metric) associated with each link in the AS3 108.

During the first communication session with the ASBR3 122, the example topology and route collector 204 also requests that the ASBR3 122 transmit external network routing information identifying external routes that are advertised by the border routers (e.g., the ASBR3 122, the ASBR4 124) of the second autonomous system AS2 108 (also block 606). During the second communication session with the ASBR5 126, the example topology and route collector 204 also requests that the ASBR5 126 transmit external network routing information identifying external routes that are advertised by the border routers (e.g., the ASBR5 126, the ASBR6 128 and the PE2 132) of the third autonomous system AS3 132 (also block 606).

The example topology and route collector 204 stores the second and the third topology information as a topology database in the example topology database storage 206 (block 608) and stores the external routes in the example external route database (also block 608).

In some examples, the example virtual positioner 210 of the example virtual route reflector 110 uses the second topology information to select an internal node (in this example IN5 136A) residing within the example second autonomous system AS2 106 (block 610). The location of the selected node IN5 136A within the AS1 104 will be used as the virtual position of the virtual route reflector 110 to select paths as described in greater detail below.

Additionally, the example path selector 212 uses the third topology information stored in the topology database storage 206 (see FIG. 2) to identify and select an internal node in the AS3 108, such as the IN12 138D (block 612). Next, the path selector 212 uses any desired method including, for example, Dijkstra's algorithm to select a “best” path from the virtual position (e.g., from the IN5 136A) to the IN12 138D located in the AS3 108. In some such examples, the best path is selected as the path from the virtual position to the IN12 138D having a lowest overall cost as compared to other possible paths between the virtual position and the IN12 138D. In some such examples, the costs associated with any links that, together, form a path are combined to formulate an accumulated IGP cost (also known as an AIGP cost). Example techniques that can be used to obtain an AIGP cost for a path having links from more than a single autonomous system are described in a Request for Comment no. 7311 entitled, “The Accumulated IGP Metric Attribute for BGP” published by the Internet Engineering Task Force (IETF). In some examples, the path selector calculates an AIGP cost for each possible path between the virtual position and the IN12 138D and then selects the path having the lowest cost.

After identifying the shortest path, the path selector 212 stores information identifying the shortest path and further identifying the AIGP cost associated with the shortest path in the example path storage 214 (block 614). In some examples, the information identifying the shortest path (i.e., the selected path) includes the address of the autonomous system boundary router that lies along the selected path. Thus, for example, assuming that a first path path1 (see FIG. 3) extending from the virtual position (e.g., the IN5 136A) to the destination node (e.g., IN12 138D) is the shortest path between the two nodes and, therefore, is the selected path, the ASBR5 126 is identified as the autonomous system boundary router that lies along the selected path. As a result, the address of the ASBR5 126 is stored with the information identifying the selected path in the path storage 214 (block 616). Additionally, information identifying the address of the source of the selected path (in this example, the IN5 136A) and the destination of the selected path (in this example, the IN12 138D) is also included in the selected path information.

If needed, the example route advertiser 216 then converts the information identifying the selected path into a route using a routing protocol or format (e.g., BGP-LS with the AIGP attribute enabled) that is suitable for transmission to the ASBR3 122 and to the ASBR5 126 (block 618). The example route advertiser 216 then provides the route information containing the selected path to the example network interface 202 for transmission to the ASBR3 122 and the ASBR5 126 (block 620) and the method ends (block 622).

Upon receipt of the advertised route, the ASBR3 122 supplies the route to the internal node IN5 136A of the AS2 106 for use in reaching the internal node IN12 138D of the AS3 108. Likewise, the ASBR5 126 supplies the route to the internal node IN12 138D of the AS3 108 for use in reaching the internal node IN5 136A of the AS2 106.

In some examples, the method represented by the program 600 is repeated until a “best” path between each internal node residing in the AS2 106 and each internal node residing in the AS3 108 has been selected and information identifying the best paths has been transmitted to the corresponding autonomous systems. Thus, the method represented by the program 600 can be used to improve routing efficiency between two autonomous systems that use the same IGP or different IGPs provided that when the IGPs used by the two autonomous system are different, the administrator takes measures to ensure that the metrics used by the IGPs are compatible, or, if needed, converts the metrics used by the IGPs to be compatible.

Referring now to FIG. 7, a program 700 represents a method by which the example virtual route reflector 110 performs route reflection for an autonomous system (e.g., the AS1 104) from a location outside of the AS1 104 by virtually positioning itself at a location within the AS1 104. As described below, in the method represented by the program 700, the virtual route reflector 110 uses a first topology describing the topology of the AS1 104, a second topology information describing the topology of the AS2 106, and a third topology information describing the topology of the AS3 108 to determine a best path between an internal node (e.g., the IN1 134A) of the AS1 104 and a provider edge router associated with the AS3 108. Referring also to FIG. 1 and FIG. 2, the method 700 begins at a block 702 after which the example network interface 202 (see FIG. 2) of the example virtual route reflector 110 (see FIG. 1 and FIG. 2) initiates a first BGP communication session with the example autonomous system boundary router ASBR1 114, and further initiates second and third BGP communication sessions with the example autonomous system boundary router ASBR3 122 (see FIG. 1) of the AS2 106 and the example autonomous system boundary router ASBR5 126 (see FIG. 1) of the AS3 108 (block 604). During the first BGP communication session, the example topology and route collector 204 (see FIG. 2) requests first topology information for the AS1 104 from the ASBR3 122 and, during the second BGP communication session, the topology and route collector 204 requests second topology information for the AS2 106 from the ASBR 3 122. During the third BGP communication session, the topology and route collector 204 requests third topology information for the AS3 108 from the ASBR5 126 (block 606). In some examples, the virtual route reflector 110 is unable to directly communicate with the ASBR5 126. In some such examples, the virtual route reflector 110 instructs the ASBR3 122 to request the third topology information from the ASBR5 126.

Responsive to the request, the ASBR1 114, the ASBR3 122 and the ASBR5 126 respond by supplying the first, second and third topology information, respectively, in a format that is transferrable using an EBGP such as, for example BGP-LS.

In some examples, the topology and route collector 204 uses the first, second, and third topology information, respectively, to identify the nodes 134 residing in the AS1 104 and the links by which the nodes 134 are coupled, the nodes 136 residing in the AS2 106 and the links by which the nodes 136 are coupled and the nodes 138 residing the AS3 108 and the links by which the nodes are coupled, respectively. The topology and route collector 204 also uses the first, second, and third topology information to identify a cost (or metric) associated with each link in the AS1 104, the AS2 106 and AS3 108.

During the first, second and third communication sessions, respectively, the example topology and route collector 204 also 1) requests that the ASBR1 114 transmit external network routing information identifying external routes that are advertised by the border routers associated with the first autonomous system AS1 104, 2) requests that the ASBR1 114 transmit external network routing information identifying external routes that are advertised by the border routers associated with the second autonomous system AS2 106, and 3) requests that the ASBR5 126 transmit external network routing information identifying external routes that are advertised by the border routers associated with the third autonomous system AS3 108 (also block 706).

The example topology and route collector 204 stores the first, second and the third topology information as a topology database in the example topology database storage 206 (block 708) and stores the external routes in the example external route database (also block 708).

In some examples, the example virtual positioner 210 of the example virtual route reflector 110 uses the first topology information to select an internal node (in this example IN1 134A) residing within the example first autonomous system AS1 104 (block 810). The location of the selected node IN1 134A within the AS1 104 will be used as the virtual position of the virtual route reflector 110 to select paths as described in greater detail below.

Additionally, the example path selector 212 uses the third topology information stored in the topology database storage 206 (see FIG. 2) to identify and selects a target network destination that is external to the first autonomous system AS1 104 (block 812). In this example, the provider edge router PE2 132 is selected as the target network destination. Next, the path selector 212 uses any desired method including, for example, Dijkstra's algorithm to select a “best” path from the virtual position (e.g., from the IN1 134A) to the PE2 132 coupled to the AS3 108. In some such examples, the best path is selected as the path extending from the virtual position (IN1 134A) to the PE2 132 that has a lowest overall cost as compared to other possible paths between the virtual position (IN1 134A) and the PE2 132. In some such examples, the costs associated with any links that, together, form a path are combined to formulate an accumulated IGP cost (also known as an AIGP cost). Example techniques that can be used to obtain an AIGP cost for a path having links from more than a single autonomous system are described in a Request for Comment no. 7311 entitled, “The Accumulated IGP Metric Attribute for BGP” published by the Internet Engineering Task Force (IETF). In some examples, the path selector calculates an AIGP cost for each possible path between the virtual position and the IN12 138D and then selects the path having the lowest cost. For illustrative purposes, the “path2” representing by the dotted line (extending from IN1 134A to ASBR1 114 and then to ASBR4 124 and then to ASBR6 128 and then to PE2 132 in FIG. 1) is determined to be the shortest path and, as such, is selected by the path selector 212. In some such examples, the overall cost of the path2 (e.g., the AIGP for path2) is determined by adding: 1) a first cost associated with the portion of the path2 extending from the PE2 132 to the ASBR6 128, 2) a second cost associated with the portion of the path2 extending from the ASBR6 128 to the ASBR4 124, and a 3) a third cost associated with the portion of the path2 extending from the ASBR1 114 to the IN1 134A residing in the first autonomous system AS1 104. There is no cost metric associated with the portion of the path2 that extends from the ASBR4 124 to the ASBR1 114 because that portion of the path2 is not associated with an IGP.

After identifying the path2 as the shortest path, the path selector 212 stores information identifying the path2 and further identifying the AIGP cost associated with the path2 in the example path storage 214 (block 814). In some examples, the information identifying the shortest path (i.e., the selected path) includes the address of the autonomous system boundary router that is on the path2 and that is nearest to the source node of the path2 (in this example IN1 134A) (block 816). Additionally, information identifying the address of the source of the selected path (in this example, the IN1 134A) and the destination of the selected path (in this example, the PE2 132) is also included in the stored path information.

If needed, the example route advertiser 216 then converts the information identifying the selected path into a route using a routing protocol or format (e.g., BGP-LS with the AIGP attribute enabled) that is suitable for transmission to the ASBR3 122 and to the ASBR5 126 (block 818). The example route advertiser 216 then provides the route information containing the selected path to the example network interface 202 for transmission to the ASBR1 114 (block 620) and the method ends (block 622).

Upon receipt of the advertised route, the ASBR 114 supplies the route to the internal node IN1 134A of the AS1 104 for use in reaching the PE2 132 associated with the example third autonomous system AS3 108.

In some examples, the method represented by the program 700 is repeated until a “best” path between each internal node residing in the AS1 104 and each internal node residing in the AS3 108 (as well as each of the boundary routers associated with the third autonomous system AS3 108) have been selected and information identifying the best paths has been transmitted to the corresponding autonomous systems. Thus, the method represented by the program 700 can be used to improve routing efficiency between three autonomous systems that use the same IGP or different IGPs provided that when the IGPs used by the three autonomous system are different, the administrator takes measures to ensure that the metrics used by the IGPs are compatible, or, if needed, converts the metrics used by the IGPs to be compatible.

FIG. 8 is a block diagram of an example processor platform 1000 capable of executing the instructions of FIGS. 4, 5, 6, and 7 to implement the virtual route reflector 110 of FIG. 2. The processor platform 800 can be, for example, a server, a personal computer, a mobile device or any other type of computing device.

The processor platform 800 of the illustrated example includes a processor 812. The processor 812 of the illustrated example is hardware. For example, the processor 812 can be implemented by one or more integrated circuits, logic circuits, microprocessors or controllers from any desired family or manufacturer. In the illustrated example of FIG. 8, the processor 812 includes one or more example processing cores 815 configured via example instructions 1032, which include the example instructions of FIGS. 4, 5, 6 and/or 7, to implement the example topology and route collector 204, the example virtual positioner 210, the example path selector 212, and the example route advertiser 216 of FIG. 2.

The processor 812 of the illustrated example includes a local memory 813 (e.g., a cache). The processor 812 of the illustrated example is in communication with a main memory including a volatile memory 814 and a non-volatile memory 816 via a bus 818. The volatile memory 814 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/or any other type of random access memory device. The non-volatile memory 816 may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory 814, 816 is controlled by a memory controller.

The processor platform 800 of the illustrated example also includes an interface circuit 820. The interface circuit 820 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and/or a PCI express interface. In the illustrated example of FIG. 8, the interface circuit 820 is also structured to implement the example network interface 202.

In the illustrated example, one or more input devices 822 are connected to the interface circuit 820. The input device(s) 822 permit(s) a user to enter data and commands into the processor 812. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and/or a voice recognition system.

One or more output devices 824 are also connected to the interface circuit 820 of the illustrated example. The output devices 824 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a printer and/or speakers). The interface circuit 820 of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip or a graphics driver processor.

The interface circuit 820 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem and/or network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 826 (e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform 800 of the illustrated example also includes one or more mass storage devices 828 for storing software and/or data. Examples of such mass storage devices 828 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.

The coded instructions 832 of FIGS. 4, 5, 6, and 7 may be stored in the mass storage device 828, in the volatile memory 814, in the non-volatile memory 816, and/or on a removable tangible computer readable storage medium such as a CD or DVD. In some examples, the mass storage device 830 may implement the example topology database storage 206 and/or the example external routes database storage 208 and/or the example path storage 214. Additionally or alternatively, in some examples the volatile memory 818 may implement the example topology database storage 206 and/or the example external routes database storage 208 and/or the example path storage 214.

From the foregoing, it will be appreciated that the above disclosed methods, apparatus and articles of manufacture permit the virtualization of route reflectors thereby saving on cost and complexity. Further, the virtual route reflectors disclosed herein can be located anywhere even, geographically distant from the autonomous system it serves and yet still effectively perform hot potato routing. Additionally, the virtual route reflectors disclosed herein are capable of performing more efficient routing of messages between two and even three autonomous systems that operate using different interior gateway protocols.

Although certain example methods, apparatus and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus and articles of manufacture fairly falling within the scope of the claims of this patent.

Briefly, one or more aspects of the technology described herein are generally directed towards adapting the multiple antenna transmission scheme for uplink data transmission in wireless communication systems based on a Doppler metric of a user equipment, which in general thereby improves the uplink MIMO (multiple input, multiple output) performance.

One or more aspects of the technology described herein comprise having a network node (network device) determine the Doppler Metric of the user equipment, and based on the Doppler metric versus a Doppler metric threshold value, inform the user equipment to change the transmission scheme, if needed. This may operate to change the transmission scheme from closed loop MIMO to open loop MIMO (Rank-1 precoder cycling), or from open loop MIMO to closed loop MIMO.

To this end, the network device operates by obtaining information about the Doppler metric of the UE, and determining if the Doppler metric is above or below a pre-defined threshold Doppler metric value. In general, if the Doppler metric is above the threshold Doppler metric value and the closed loop MIMO transmission scheme is in use, the network device communicates a recommendation to the usual equipment to switch to the Rank-1 precoder cycling transmission scheme. Conversely, if the Doppler metric is below the threshold Doppler metric value and the transmission scheme in use is the Rank-1 precoder cycling transmission scheme, the network device communicates a recommendation to the user equipment to switch to the closed loop MIMO transmission scheme. Note that alternatively, instead of communicating a recommendation, the network device can switch off the transmit precoding matrix index (TPMI) information, and notify the user equipment of this change.

Operations for which the user equipment is responsible include receiving the recommendation from the network device, and determining the transmission scheme based on the network device recommendation. Further, the user equipment applies the recommended transmission scheme for uplink MIMO data transmission.

As can be readily appreciated, with the above technology, the uplink MIMO performance is improved at high Doppler frequencies. This results in increased network capacity due to improved user equipment performance at high Doppler frequencies.

It should be understood that any of the examples and terms used herein are non-limiting. For instance, the examples are based on New Radio (NR, sometimes referred to as 5G) communications between a user equipment exemplified as a smartphone or the like and network device; however virtually any communications devices may benefit from the technology described herein, and/or their use in different spectrums may likewise benefit. Further, as used in the examples herein, closed loop MIMO data transmission and Rank-1 precoder cycling transmission schemes are described as non-limiting examples; however, it is understood that the technology described herein can be used to adaptively switch between one or more other transmission schemes. Thus, any of the embodiments, aspects, concepts, structures, functionalities, or examples described herein are non-limiting, and the technology may be used in various ways that provide benefits and advantages in radio communications in general.

In some embodiments the non-limiting term “radio network node” or simply “network node,” “radio network device or simply “network device” is used herein. These terms may be used interchangeably, and refer to any type of network node that serves user equipment and/or connected to other network node or network element or any radio node from where user equipment receives signal. Examples of radio network nodes are Node B, base station (BS), multi-standard radio (MSR) node such as MSR BS, gNodeB, eNode B, network controller, radio network controller (RNC), base station controller (BSC), relay, donor node controlling relay, base transceiver station (BTS), access point (AP), transmission points, transmission nodes, RRU, RRH, nodes in distributed antenna system (DAS) etc.

In some embodiments the non-limiting term user equipment (UE) is used. It refers to any type of wireless device that communicates with a radio network node in a cellular or mobile communication system. Examples of user equipment are target device, device to device (D2D) user equipment, machine type user equipment or user equipment capable of machine to machine (M2M) communication, PDA, Tablet, mobile terminals, smart phone, laptop embedded equipped (LEE), laptop mounted equipment (LME), USB dongles etc.

FIG. 1 illustrates an example wireless communication system 100 in accordance with various aspects and embodiments of the subject technology. In one or more embodiments, the system 100 can comprise one or more user equipment UEs 102(1)-102(n).

In various embodiments, the system 100 is or comprises a wireless communication network serviced by one or more wireless communication network providers. In example embodiments, a UE 102 can be communicatively coupled to the wireless communication network via a network device 104 (e.g., network node). The network device 104 can communicate with the user equipment (UE) 102, thus providing connectivity between the UE and the wider cellular network.

In example implementations, each UE 102 such as the UE 102(1) is able to send and/or receive communication data via a wireless link to the network device 104. The dashed arrow lines from the network device 104 to the UE 102 represent downlink (DL) communications and the solid arrow lines from the UE 102 to the network devices 104 represents uplink (UL) communications.

The system 100 can further include one or more communication service provider networks 106 that facilitate providing wireless communication services to various user equipment, including UEs 102(1)-102(n), via the network device 104 and/or various additional network devices (not shown) included in the one or more communication service provider networks 106. The one or more communication service provider networks 106 can include various types of disparate networks, including but not limited to: cellular networks, femto networks, picocell networks, microcell networks, internet protocol (IP) networks Wi-Fi service networks, broadband service network, enterprise networks, cloud based networks, and the like. For example, in at least one implementation, system 100 can be or include a large scale wireless communication network that spans various geographic areas. According to this implementation, the one or more communication service provider networks 106 can be or include the wireless communication network and/or various additional devices and components of the wireless communication network (e.g., additional network devices and cell, additional UEs, network server devices, etc.).

The network device 104 can be connected to the one or more communication service provider networks 106 via one or more backhaul links 108. For example, the one or more backhaul links 108 can comprise wired link components, such as a T1/E1 phone line, a digital subscriber line (DSL) (e.g., either synchronous or asynchronous), an asymmetric DSL (ADSL), an optical fiber backbone, a coaxial cable, and the like. The one or more backhaul links 108 can also include wireless link components, such as but not limited to, line-of-sight (LOS) or non-LOS links which can include terrestrial air-interfaces or deep space links (e.g., satellite communication links for navigation).

Some embodiments are described in particular for 5G new radio systems. The embodiments are however applicable to any radio access technology (RAT) or multi-RAT system where the user equipment operates using multiple carriers e.g. LTE FDD/TDD, WCMDA/HSPA, GSM/GERAN, Wi Fi, WLAN, WiMax, CDMA2000 etc.

The embodiments are applicable to single carrier as well as to multicarrier (MC) or carrier aggregation (CA) operation of the user equipment. The term carrier aggregation (CA) is also called (e.g. interchangeably called) “multi-carrier system”, “multi-cell operation”, “multi-carrier operation”, “multi-carrier” transmission and/or reception.

Note that the solutions outlined equally applies for Multi RAB (radio bearers) on some carriers (that is data plus speech is simultaneously scheduled). Some embodiments are described in particular for 5G new radio systems. The embodiments are however applicable to any radio access technology (RAT) or multi-RAT system where the user equipment operates using multiple carriers e.g. LTE FDD/TDD, WCMDA/HSPA, GSM/GERAN, Wi Fi, WLAN, WiMax, CDMA2000 etc.

The embodiments are applicable to single carrier as well as to multicarrier (MC) or carrier aggregation (CA) operation of the user equipment. The term carrier aggregation (CA) is also called (e.g. interchangeably called) “multi-carrier system”, “multi-cell operation”, “multi-carrier operation”, “multi-carrier” transmission and/or reception.

Note that the solutions outlined equally applies for Multi RAB (radio bearers) on some carriers (that is, data plus speech is simultaneously scheduled).

FIG. 1 illustrates an example wireless communication system 100 in accordance with various aspects and embodiments of the subject technology. In one or more embodiments, the system 100 can comprise one or more user equipment UEs 102(1)-102(n).

In various embodiments, the system 100 is or comprises a wireless communication network serviced by one or more wireless communication network providers. In example embodiments, a UE 102 can be communicatively coupled to the wireless communication network via a network device 104 (e.g., network node). The network device 104 can communicate with the user equipment (UE) 102, thus providing connectivity between the UE and the wider cellular network.

In example implementations, each UE 102 such as the UE 102(1) is able to send and/or receive communication data via a wireless link to the network device 104. The dashed arrow lines from the network device 104 to the UE 102 represent downlink (DL) communications and the solid arrow lines from the UE 102 to the network devices 104 represents uplink (UL) communications.

The system 100 can further include one or more communication service provider networks 106 that facilitate providing wireless communication services to various user equipment, including UEs 102(1)-102(n), via the network device 104 and/or various additional network devices (not shown) included in the one or more communication service provider networks 106. The one or more communication service provider networks 106 can include various types of disparate networks, including but not limited to: cellular networks, femto networks, picocell networks, microcell networks, internet protocol (IP) networks Wi-Fi service networks, broadband service network, enterprise networks, cloud based networks, and the like. For example, in at least one implementation, system 100 can be or include a large scale wireless communication network that spans various geographic areas. According to this implementation, the one or more communication service provider networks 106 can be or include the wireless communication network and/or various additional devices and components of the wireless communication network (e.g., additional network devices and cell, additional UEs, network server devices, etc.).

The network device 104 can be connected to the one or more communication service provider networks 106 via one or more backhaul links 108. For example, the one or more backhaul links 108 can comprise wired link components, such as a T1/E1 phone line, a digital subscriber line (DSL) (e.g., either synchronous or asynchronous), an asymmetric DSL (ADSL), an optical fiber backbone, a coaxial cable, and the like. The one or more backhaul links 108 can also include wireless link components, such as but not limited to, line-of-sight (LOS) or non-LOS links which can include terrestrial air-interfaces or deep space links (e.g., satellite communication links for navigation).

The wireless communication system 100 can employ various cellular systems, technologies, and modulation schemes to facilitate wireless radio communications between devices (e.g., the UE 102 and the network device 104). While example embodiments might be described for 5G new radio (NR) systems, the embodiments can be applicable to any radio access technology (RAT) or multi-RAT system where the UE operates using multiple carriers e.g. LTE FDD/TDD, GSM/GERAN, CDMA2000 etc. For example, the system 100 can operate in accordance with global system for mobile communications (GSM), universal mobile telecommunications service (UMTS), long term evolution (LTE), LTE frequency division duplexing (LTE FDD, LTE time division duplexing (TDD), high speed packet access (HSPA), code division multiple access (CDMA), wideband CDMA (WCMDA), CDMA2000, time division multiple access (TDMA), frequency division multiple access (FDMA), multi-carrier code division multiple access (MC-CDMA), single-carrier code division multiple access (SC-CDMA), single-carrier FDMA (SC-FDMA), orthogonal frequency division multiplexing (OFDM), discrete Fourier transform spread OFDM (DFT-spread OFDM) single carrier FDMA (SC-FDMA), Filter bank based multi-carrier (FBMC), zero tail DFT-spread-OFDM (ZT DFT-s-OFDM), generalized frequency division multiplexing (GFDM), fixed mobile convergence (FMC), universal fixed mobile convergence (UFMC), unique word OFDM (UW-OFDM), unique word DFT-spread OFDM (UW DFT-Spread-OFDM), cyclic prefix OFDM CP-OFDM, resource-block-filtered OFDM, Wi Fi, WLAN, WiMax, and the like. However, various features and functionalities of system 100 are particularly described wherein the devices (e.g., the UEs 102 and the network device 104) of system 100 are configured to communicate wireless signals using one or more multi carrier modulation schemes, wherein data symbols can be transmitted simultaneously over multiple frequency subcarriers (e.g., OFDM, CP-OFDM, DFT-spread OFMD, UFMC, FMBC, etc.). The embodiments are applicable to single carrier as well as to multicarrier (MC) or carrier aggregation (CA) operation of the UE. The term carrier aggregation (CA) is also called (e.g. interchangeably called) “multi-carrier system”, “multi-cell operation”, “multi-carrier operation”, “multi-carrier” transmission and/or reception. Note that some embodiments are also applicable for Multi RAB (radio bearers) on some carriers (that is data plus speech is simultaneously scheduled).

In various embodiments, the system 100 can be configured to provide and employ 5G wireless networking features and functionalities. With 5G networks that may use waveforms that split the bandwidth into several sub bands, different types of services can be accommodated in different sub bands with the most suitable waveform and numerology, leading to improved spectrum utilization for 5G networks. Notwithstanding, in the mmWave spectrum, the millimeter waves have shorter wavelengths relative to other communications waves, whereby mmWave signals can experience severe path loss, penetration loss, and fading. However, the shorter wavelength at mmWave frequencies also allows more antennas to be packed in the same physical dimension, which allows for large-scale spatial multiplexing and highly directional beamforming.

Performance can be improved if both the transmitter and the receiver are equipped with multiple antennas. Multi-antenna techniques can significantly increase the data rates and reliability of a wireless communication system. The use of multiple input multiple output (MIMO) techniques, which was introduced in the third-generation partnership project (3GPP) and has been in use (including with LTE), is a multi-antenna technique that can improve the spectral efficiency of transmissions, thereby significantly boosting the overall data carrying capacity of wireless systems. The use of multiple-input multiple-output (MIMO) techniques can improve mmWave communications; MIMO can be used for achieving diversity gain, spatial multiplexing gain and beamforming gain.

Note that using multi-antennas does not always mean that MIMO is being used. For example, a configuration can have two downlink antennas, and these two antennas can be used in various ways. In addition to using the antennas in a 2×2 MIMO scheme, the two antennas can also be used in a diversity configuration rather than MIMO configuration. Even with multiple antennas, a particular scheme might only use one of the antennas (e.g., LTE specification's transmission mode 1, which uses a single transmission antenna and a single receive antenna). Or, only one antenna can be used, with various different multiplexing, precoding methods etc.

The MIMO technique uses a commonly known notation (M×N) to represent MIMO configuration in terms number of transmit (M) and receive antennas (N) on one end of the transmission system. The common MIMO configurations used for various technologies are: (2×1), (1×2), (2×2), (4×2), (8×2) and (2×4), (4×4), (8×4). The configurations represented by (2×1) and (1×2) are special cases of MIMO known as transmit diversity (or spatial diversity) and receive diversity. In addition to transmit diversity (or spatial diversity) and receive diversity, other techniques such as spatial multiplexing (comprising both open-loop and closed-loop), beamforming, and codebook-based precoding can also be used to address issues such as efficiency, interference, and range.

In FIG. 1, as described herein, a user equipment (e.g., 102(1)) is configured to provide Doppler-related data 110, and may receive a transmission scheme recommendation 112, (or a notification that results in a change to the transmission scheme). To this end, the user equipment transmits information from which a Doppler metric may be computed, and based in part on the computed Doppler metric, the network device 104 can decide whether the user equipment is to change its transmission scheme.

FIG. 2 initially shows a state diagram having two exemplified states for transmission schemes, namely a closed loop MIMO transmission scheme state 222 and a Rank-1 precoder cycling transmission scheme state 224. As can be seen, if the user equipment is operating in the closed loop MIMO scheme state 222 and a high Doppler (relative to a threshold value) is determined, the state transitions to the Rank-1 precoder cycling transmission scheme state 224. Conversely, if the user equipment is operating in the Rank-1 precoder cycling transmission scheme state 224 and a low Doppler (relative to a threshold value) is determined, the state of the transmission scheme in use transitions to the closed loop MIMO scheme state 222.

To this end, in one or more implementations, when the network device 104 detects the user equipment 102 is moving with a high Doppler frequency greater than a threshold D_(th), the network device 104 communicates to the user equipment 102 to change to rank-1 precoder cycling, e.g., by using one bit information in the downlink control channel. With rank-1 precoder cycling, the user equipment 102 can use random precoders at the transmission side. The random precoders are done at the PRG, (i.e., block of resource block groups). In general, with the rank-1 random precoding, with the rank equal to one, the reliability of the transmitted signal increases, thereby reducing the CSI (Channel State Information) estimation error due to the high Doppler shift between the transmitter and the receiver. Note that for Rank-1 precoder cycling, instead of only one SRS (sounding reference signals, specifically intended to be used by the network device to acquire CSI and beam specific-information), the user equipment 102 sends multiple SRS signals, e.g., N, where each SRS signal is precoded with different precoder cycling, e.g., the precoders set is M entries. Then a first SRS configuration uses precoder cycling starting from 1 . . . M. The second SRS source starts the precoders cycling from 2 . . . M.1. The third SRS resource starts the precoder cycling from 3 . . . M.1.2. etc.

Once the network device 104 receives these precoded SRS, the network device 104 estimates the SINR/CQI/MCS (signal-to-interference plus noise ratio/channel quality indicator/modulation and coding scheme) of each SRS and chooses the best SRS source indication (SRI), and indicates this information in the downlink control channel. Once the user equipment 102 receives the SRI, the user equipment 102 uses the same precoder cycling, that is, selected for SRI, and uses the same precoder cycling for data transmission (PUSCH).

Similarly, whenever the network device 104 detects the user equipment has changed its speed and is moving with a relatively slow speed (based on the Doppler frequency), the network device 104 will inform the user equipment 102 to change the transmission scheme to the closed loop MIMO mode.

FIG. 3 is a graph representation 330 that exemplifies the spectral efficiency of the rank-1 precoder cycling as a function of Doppler frequency with wideband CQI, in contrast to that of closed-loop MIMO. It can be observed from FIG. 3 that while closed loop MIMO the grades in spectral efficiency as the Doppler frequency increases, the Rank-1 precoder cycling performance varies very little.

In this particular example, the network device 104 thus informs the user equipment 102 to switch to Rank-1 precoder cycling when the user equipment's Doppler frequency goes above 320. Note however that it is feasible to anticipate an increase in the user equipment Doppler frequency, and for example, use a different threshold value as speed is increasing. Similarly, the network device 104 informs the user equipment 102 to switch to the closed loop MIMO transmission scheme as the user equipment speed, corresponding to the Doppler frequency, slows down. Again, the network device 104 can recognize that the user equipment 102 is slowing down and use a different threshold value in this situation. In this way, a user equipment with a Doppler frequency varying around 320 does not result in frequent changes to the transmission scheme. As another alternative, regardless of whether the user equipment is slowing down or speeding up, different threshold values can be used for what is considered a high Doppler frequency before changing to the Rank-1 precoder cycling transmission scheme versus what is considered a low Doppler frequency before changing to the closed loop MIMO transmission scheme for generally the same reason, e.g. to avoid too-frequent changes.

FIG. 4 shows an example of one set of operations using the Doppler metric data as the decision criterion for switching to precoder cycling. Note that other decision criterion/criteria in addition to the Doppler metric data may be used for making a decision.

Operation 402 represents obtaining at the network device 104 the Doppler metric for a specific user equipment, e.g., 102(1). Note that this may include computing the Doppler metric from one or more received data.

By way of some non-limiting examples, direct speed measurement is one way the Doppler metric may be obtained, e.g., the network device 104 can compute the direct speed of the user equipment, such as by obtaining positioning information or GPS information at multiple intervals. Then the Doppler metric (D_(m)) can be taken as average of the individual speed measurement. Another way is to obtain/determine a rate of change of uplink channel estimates, e.g., the network device 104 estimates the uplink channel. The rate of change of uplink channel gives a measure of Doppler metric D_(m).

Yet another way to obtain the Doppler metric D_(m) is based on the rate of change of the downlink channel quality information (CQI):

-   -   Let CQI represent the channel quality information reported by         the user equipment at any given time interval.     -   Let ΔCQI represents the rate of change of CQI over K.     -   Then the Doppler metric can be computed as D_(m)=ΔCQI/ΔT

Once D_(m) is obtained, operation 404 determines if the user equipment is moving a high speed (high Doppler) or low speed (low Doppler) based on the Doppler threshold value D_(th). If a low speed, operation 406 represents continuing to use/switching to the closed loop MIMO scheme. If at a high speed, operation 408 represents informing the UE to change to Rank-1 precoding cycling (or continuing to use Rank-1 precoding cycling if already using). Once the UE receives the information, the UE changes its transmission scheme (operation 410) if so informed, e.g., switches to rank-1 precoder cycling or closed loop MIMO if needed, and transmits data.

In an alternative embodiment, FIG. 5 describes a scenario in which the network device 104 can obtain and check the Doppler metric at steps 502 and 504 (as in FIG. 4, steps 402 and 404), and if above the threshold, inform the user equipment (operation 508) that the network device switches off the TPMI indication in the downlink control channel. This is because without TPMI and TRI=1, (where TRI is the transmit rank indicator), the network reports the CQI which corresponds to the CQI of precoded SRS. In this way, the user equipment can use precoder cycling as described herein. Note that in the above technique the network needs to inform the user equipment (operation 508) that it switches off the TPMI indication. This can be done, for example, by having the network uses higher layer signaling (RRC) or physical layer signaling (DCI) to inform the user equipment. As can be readily appreciated, operation 506 can do nothing if the scheme is the appropriate one given the Doppler metric versus the threshold value, or can switch/notify that TPMI is back on. Once the UE receives the information, the UE changes its transmission scheme (operation 510) if so informed, e.g., switches to rank-1 precoder cycling or closed loop MIMO if needed, and transmits data.

It yet another alternative embodiment, FIG. 6 shows a scenario in which the user equipment changes based on its own evaluation of the Doppler metric versus the threshold value. Note that operation 602 represents the user equipment obtaining the Doppler metric, such as by computing via GPS coordinate changes over time. Notwithstanding, it is feasible for the network device to provide at least some information used in determining the Doppler metric.

Operation 604 evaluates the Doppler metric versus the threshold value. If below, the user equipment switches (or continues to use) the closed loop MIMO transmission scheme, as represented via operation 606. If above the threshold, the user equipment switches (or continues to use) the Rank-1 precoder cycling scheme, as represented via operation 608.

One or more aspects, generally represented in FIG. 7, are generally directed towards obtaining, by a network device comprising a processor, Doppler metric data corresponding to a relative speed of a user equipment as represented via operation 702. Operation 704 represents evaluating, by the network device, the Doppler metric data with respect to a threshold value. Operation 706 represents, based at least in part on a result of the evaluating with respect to the threshold value, taking, by the network device, an action to change a transmission scheme of the user equipment according to a change of a group of changes, wherein the group of changes comprises a first change from a closed loop multiple input multiple output transmission scheme to a rank-1 precoder cycling transmission scheme and a second change from the rank-1 precoder cycling transmission scheme to the closed loop multiple input multiple output transmission scheme.

The result of the evaluating can indicate that the Doppler metric data is above the threshold value, and taking the action can comprise communicating a recommendation of the transmission scheme to the user equipment to change the transmission scheme according to the first change from the closed loop multiple input multiple output transmission scheme to the rank-1 precoder cycling transmission scheme.

The result of the evaluating can indicate that the Doppler metric data is below the threshold value, and taking the action can comprise communicating a recommendation of the transmission scheme to the user equipment to change the transmission scheme according to the second change from the rank-1 precoder cycling transmission scheme to the closed loop multiple input multiple output transmission scheme.

Taking the action can comprise communicating a recommendation of the transmission scheme to the user equipment via a flag value in downlink control channel data.

The result of the evaluating can indicate that the Doppler metric data is above the threshold value, and taking the action can comprise communicating information to the user equipment that informs the user equipment that the network device is discontinuing a process of sending a transmit precoding matrix index indication to the user equipment. Communicating the information can comprise communicating the information via radio resource control signaling or physical layer signaling.

The result of the evaluating can indicate that the Doppler metric data is above the threshold value, and taking the action can comprise communicating information to the user equipment that informs the user equipment that the network device is stopping a process of sending transmit rank information to the user equipment.

The result of the evaluating can indicate that the Doppler metric data is below the threshold value, and taking the action can comprise communicating information to the user equipment that informs the user equipment that the network device is at least one of starting a first process of sending a transmit precoding matrix index indication to the user equipment, or resuming a second process of sending transmit rank information to the user equipment.

Obtaining the Doppler metric data corresponding to the relative speed of the user equipment can comprise obtaining positioning data at intervals, and determining the Doppler metric data based on the positioning data. Obtaining the Doppler metric data corresponding to the relative speed of the user equipment can comprise obtaining a rate of change based on uplink channel estimates, and obtaining the Doppler metric data based on the rate of change. Obtaining the Doppler metric data corresponding to the relative speed of the user equipment can comprise obtaining reports comprising respective downlink channel quality information, determining a rate of change of the respective downlink channel quality information of the reports over time, and determining the Doppler metric data as a function of the rate of change of the respective downlink channel quality information over time.

Further aspects may comprise selecting, by the network device, a first value as the threshold value in response to a change in the Doppler metric data being determined to correspond to an increase in the relative speed of the user equipment, and selecting, by the network device, a second value as the threshold value in response to the change in the Doppler metric data being determined to correspond to a decrease in the relative speed of the user equipment.

An example embodiment of a network device 104 comprising a processor and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, is represented in FIG. 8. Example operations can include obtaining Doppler metric data corresponding to a relative speed of a user equipment (operation 802), and

In response to determining that the Doppler metric data is above a threshold value, initiating a first action to control a transmission scheme of the user equipment to be a rank-1 precoder cycling transmission scheme (operation 804). Operation 806 represents, in response to determining that the Doppler metric data is not above the threshold value, initiating a second action to control the transmission scheme of the user equipment to be a closed loop multiple input multiple output transmission scheme.

Initiating the first action to control the transmission scheme can comprise communicating a first recommendation of the rank-1 precoder cycling transmission scheme transmission scheme to the user equipment, and the initiating the second action to control the transmission scheme can comprise communicating a second recommendation of the closed loop multiple input multiple output transmission scheme to the user equipment.

Initiating the first action to control the transmission scheme can comprise communicating information to the user equipment that informs the user equipment that the network device is turning off sending transmit rank information to the user equipment, and initiating the second action to control the transmission scheme can comprise communicating other information to the user equipment that informs the user equipment that the network device is turning on the sending of the transmit rank information to the user equipment.

Obtaining the Doppler metric data corresponding to the relative speed of the user equipment can comprise at least one of: determining the Doppler metric data based on global positioning data obtained at different times, obtaining a rate of change based on uplink channel estimates, and using the rate of change to obtain the Doppler metric data, or obtaining downlink channel quality information reports, determining a rate of change of downlink channel quality information in the downlink channel quality information reports over time, and determining the Doppler metric data based on the rate of change.

Further operations can comprise, in response to a change in the Doppler metric data being determined to correspond to an increase in the relative speed of the user equipment, selecting a first value as the threshold value, and in response to the change in the Doppler metric data being determined to correspond to a decrease in the relative speed of the user equipment, selecting a second value as the threshold value.

FIG. 9 represents operations, such as in the form of a machine-readable storage medium, comprising executable instructions that, when executed by a processor of a user equipment, facilitate performance of operations. Operation 902 represents determining Doppler metric data corresponding to a speed of the user equipment relative to a network node device. Operation 904 represents, in response to determining that the Doppler metric data has transitioned a threshold Doppler metric value, selecting (operation 906) a rank-1 precoder cycling transmission protocol for use by the user equipment, and transmitting data (operation 908) from the user equipment using the rank-1 precoder cycling transmission protocol.

Further operations can comprise, in response determining that the Doppler metric data has not transitioned the threshold Doppler metric value, selecting a closed loop multiple input multiple output transmission protocol for use by the user equipment, and transmitting the data from the user equipment using the closed loop multiple input multiple output transmission protocol. Still further operations can comprise, selecting a first value as the threshold Doppler metric value in response to determining a change in the Doppler metric data over time corresponds to an increase in the relative speed of the user equipment, and selecting a second value as the threshold Doppler metric value in response to determining the change in the Doppler metric data over time corresponds to a decrease in the relative speed of the user equipment.

Referring now to FIG. 10, illustrated is an example block diagram of an example mobile handset 1000 operable to engage in a system architecture that facilitates wireless communications according to one or more embodiments described herein. Although a mobile handset is illustrated herein, it will be understood that other devices can be a mobile device, and that the mobile handset is merely illustrated to provide context for the embodiments of the various embodiments described herein. The following discussion is intended to provide a brief, general description of an example of a suitable environment in which the various embodiments can be implemented. While the description includes a general context of computer-executable instructions embodied on a machine-readable storage medium, those skilled in the art will recognize that the innovation also can be implemented in combination with other program modules and/or as a combination of hardware and software.

Generally, applications (e.g., program modules) can include routines, programs, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the methods described herein can be practiced with other system configurations, including single-processor or multiprocessor systems, minicomputers, mainframe computers, as well as personal computers, hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices.

A computing device can typically include a variety of machine-readable media. Machine-readable media can be any available media that can be accessed by the computer and includes both volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media can comprise computer storage media and communication media. Computer storage media can include volatile and/or non-volatile media, removable and/or non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, solid state drive (SSD) or other solid-state storage technology, Compact Disk Read Only Memory (CD ROM), digital video disk (DVD), Blu-ray disk, or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer. In this regard, the terms “tangible” or “non-transitory” herein as applied to storage, memory or computer-readable media, are to be understood to exclude only propagating transitory signals per se as modifiers and do not relinquish rights to all standard storage, memory or computer-readable media that are not only propagating transitory signals per se.

Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media

The handset includes a processor 1002 for controlling and processing all onboard operations and functions. A memory 1004 interfaces to the processor 1002 for storage of data and one or more applications 1006 (e.g., a video player software, user feedback component software, etc.). Other applications can include voice recognition of predetermined voice commands that facilitate initiation of the user feedback signals. The applications 1006 can be stored in the memory 1004 and/or in a firmware 1008, and executed by the processor 1002 from either or both the memory 1004 or/and the firmware 1008. The firmware 1008 can also store startup code for execution in initializing the handset 1000. A communications component 1010 interfaces to the processor 1002 to facilitate wired/wireless communication with external systems, e.g., cellular networks, VoIP networks, and so on. Here, the communications component 1010 can also include a suitable cellular transceiver 1011 (e.g., a GSM transceiver) and/or an unlicensed transceiver 1013 (e.g., Wi-Fi, WiMax) for corresponding signal communications. The handset 1000 can be a device such as a cellular telephone, a PDA with mobile communications capabilities, and messaging-centric devices. The communications component 1010 also facilitates communications reception from terrestrial radio networks (e.g., broadcast), digital satellite radio networks, and Internet-based radio services networks

The handset 1000 includes a display 1012 for displaying text, images, video, telephony functions (e.g., a Caller ID function), setup functions, and for user input. For example, the display 1012 can also be referred to as a “screen” that can accommodate the presentation of multimedia content (e.g., music metadata, messages, wallpaper, graphics, etc.). The display 1012 can also display videos and can facilitate the generation, editing and sharing of video quotes. A serial I/O interface 1014 is provided in communication with the processor 1002 to facilitate wired and/or wireless serial communications (e.g., USB, and/or IEEE 1094) through a hardwire connection, and other serial input devices (e.g., a keyboard, keypad, and mouse). This supports updating and troubleshooting the handset 1000, for example. Audio capabilities are provided with an audio I/O component 1016, which can include a speaker for the output of audio signals related to, for example, indication that the user pressed the proper key or key combination to initiate the user feedback signal. The audio I/O component 1016 also facilitates the input of audio signals through a microphone to record data and/or telephony voice data, and for inputting voice signals for telephone conversations.

The handset 1000 can include a slot interface 1018 for accommodating a SIC (Subscriber Identity Component) in the form factor of a card Subscriber Identity Module (SIM) or universal SIM 1020, and interfacing the SIM card 1020 with the processor 1002. However, it is to be appreciated that the SIM card 1020 can be manufactured into the handset 1000, and updated by downloading data and software.

The handset 1000 can process IP data traffic through the communications component 1010 to accommodate IP traffic from an IP network such as, for example, the Internet, a corporate intranet, a home network, a person area network, etc., through an ISP or broadband cable provider. Thus, VoIP traffic can be utilized by the handset 1000 and IP-based multimedia content can be received in either an encoded or a decoded format.

A video processing component 1022 (e.g., a camera) can be provided for decoding encoded multimedia content. The video processing component 1022 can aid in facilitating the generation, editing, and sharing of video quotes. The handset 1000 also includes a power source 1024 in the form of batteries and/or an AC power subsystem, which power source 1024 can interface to an external power system or charging equipment (not shown) by a power 110 component 1026.

The handset 1000 can also include a video component 1030 for processing video content received and, for recording and transmitting video content. For example, the video component 1030 can facilitate the generation, editing and sharing of video quotes. A location tracking component 1032 facilitates geographically locating the handset 1000. As described hereinabove, this can occur when the user initiates the feedback signal automatically or manually. A user input component 1034 facilitates the user initiating the quality feedback signal. The user input component 1034 can also facilitate the generation, editing and sharing of video quotes. The user input component 1034 can include such conventional input device technologies such as a keypad, keyboard, mouse, stylus pen, and/or touch screen, for example.

Referring again to the applications 1006, a hysteresis component 1036 facilitates the analysis and processing of hysteresis data, which is utilized to determine when to associate with the access point. A software trigger component 1038 can be provided that facilitates triggering of the hysteresis component 1036 when the Wi-Fi transceiver 1013 detects the beacon of the access point. A SIP client 1040 enables the handset 1000 to support SIP protocols and register the subscriber with the SIP registrar server. The applications 1006 can also include a client 1042 that provides at least the capability of discovery, play and store of multimedia content, for example, music.

The handset 1000, as indicated above related to the communications component 1010, includes an indoor network radio transceiver 1013 (e.g., Wi-Fi transceiver). This function supports the indoor radio link, such as IEEE 802.11, for the dual-mode GSM handset 1000. The handset 1000 can accommodate at least satellite radio services through a handset that can combine wireless voice and digital radio chipsets into a single handheld device.

Referring now to FIG. 11, illustrated is an example block diagram of an example computer 1100 operable to engage in a system architecture that facilitates wireless communications according to one or more embodiments described herein. The computer 1100 can provide networking and communication capabilities between a wired or wireless communication network and a server (e.g., Microsoft server) and/or communication device. In order to provide additional context for various aspects thereof, FIG. 11 and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the various aspects of the innovation can be implemented to facilitate the establishment of a transaction between an entity and a third party. While the description above is in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that the innovation also can be implemented in combination with other program modules and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the various methods can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, hand-held computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can be operatively coupled to one or more associated devices.

The illustrated aspects of the innovation can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

Computing devices typically include a variety of media, which can include computer-readable storage media or communications media, which two terms are used herein differently from one another as follows.

Computer-readable storage media can be any available storage media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media can be implemented in connection with any method or technology for storage of information such as computer-readable instructions, program modules, structured data, or unstructured data. Computer-readable storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible and/or non-transitory media which can be used to store desired information. Computer-readable storage media can be accessed by one or more local or remote computing devices, e.g., via access requests, queries, or other data retrieval protocols, for a variety of operations with respect to the information stored by the medium.

Communications media can embody computer-readable instructions, data structures, program modules, or other structured or unstructured data in a data signal such as a modulated data signal, e.g., a carrier wave or other transport mechanism, and includes any information delivery or transport media. The term “modulated data signal” or signals refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in one or more signals. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

The techniques described herein can be applied to any device or set of devices (machines) capable of running programs and processes. It can be understood, therefore, that servers including physical and/or virtual machines, personal computers, laptops, handheld, portable and other computing devices and computing objects of all kinds including cell phones, tablet/slate computers, gaming/entertainment consoles and the like are contemplated for use in connection with various implementations including those exemplified herein. Accordingly, the general-purpose computing mechanism described below with reference to FIG. 11 is but one example of a computing device.

In order to provide a context for the various aspects of the disclosed subject matter, FIG. 11, and the following discussion, are intended to provide a brief, general description of a suitable environment in which the various aspects of the disclosed subject matter can be implemented. While the subject matter has been described above in the general context of computer-executable instructions of a computer program that runs on a computer and/or computers, those skilled in the art will recognize that the disclosed subject matter also can be implemented in combination with other program modules. Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks and/or implement particular abstract data types.

In the subject specification, terms such as “store,” “storage,” “data store,” data storage,” “database,” and substantially any other information storage component relevant to operation and functionality of a component, refer to “memory components,” or entities embodied in a “memory” or components comprising the memory. It will be appreciated that the memory components described herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory, by way of illustration, and not limitation, volatile memory 1120 (see below), non-volatile memory 1122 (see below), disk storage 1124 (see below), and memory storage 1146 (see below). Further, nonvolatile memory can be included in read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). Additionally, the disclosed memory components of systems or methods herein are intended to comprise, without being limited to comprising, these and any other suitable types of memory.

Moreover, it will be noted that the disclosed subject matter can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, mini-computing devices, mainframe computers, as well as personal computers, hand-held computing devices (e.g., PDA, phone, watch, tablet computers, netbook computers, . . . ), microprocessor-based or programmable consumer or industrial electronics, and the like. The illustrated aspects can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network; however, some if not all aspects of the subject disclosure can be practiced on stand-alone computers. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

FIG. 11 illustrates a block diagram of a computing system 1100 operable to execute the disclosed systems and methods in accordance with an embodiment. Computer 1112, which can be, for example, part of the hardware of system 1120, includes a processing unit 1114, a system memory 1116, and a system bus 1118. System bus 1118 couples system components including, but not limited to, system memory 1116 to processing unit 1114. Processing unit 1114 can be any of various available processors. Dual microprocessors and other multiprocessor architectures also can be employed as processing unit 1114.

System bus 1118 can be any of several types of bus structure(s) including a memory bus or a memory controller, a peripheral bus or an external bus, and/or a local bus using any variety of available bus architectures including, but not limited to, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics, VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Firewire (IEEE 1194), and Small Computer Systems Interface (SCSI).

System memory 1116 can include volatile memory 1120 and nonvolatile memory 1122. A basic input/output system (BIOS), containing routines to transfer information between elements within computer 1112, such as during start-up, can be stored in nonvolatile memory 1122. By way of illustration, and not limitation, nonvolatile memory 1122 can include ROM, PROM, EPROM, EEPROM, or flash memory. Volatile memory 1120 includes RAM, which acts as external cache memory. By way of illustration and not limitation, RAM is available in many forms such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM).

Computer 1112 can also include removable/non-removable, volatile/non-volatile computer storage media. FIG. 11 illustrates, for example, disk storage 1124. Disk storage 1124 includes, but is not limited to, devices like a magnetic disk drive, floppy disk drive, tape drive, flash memory card, or memory stick. In addition, disk storage 1124 can include storage media separately or in combination with other storage media including, but not limited to, an optical disk drive such as a compact disk ROM device (CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RW Drive) or a digital versatile disk ROM drive (DVD-ROM). To facilitate connection of the disk storage devices 1124 to system bus 1118, a removable or non-removable interface is typically used, such as interface 1126.

Computing devices typically include a variety of media, which can include computer-readable storage media or communications media, which two terms are used herein differently from one another as follows.

Computer-readable storage media can be any available storage media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media can be implemented in connection with any method or technology for storage of information such as computer-readable instructions, program modules, structured data, or unstructured data. Computer-readable storage media can include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, solid state drive (SSD) or other solid-state storage technology, compact disk read only memory (CD ROM), digital versatile disk (DVD), Blu-ray disc or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or other tangible and/or non-transitory media which can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” herein as applied to storage, memory or computer-readable media, are to be understood to exclude only propagating transitory signals per se as modifiers and do not relinquish rights to all standard storage, memory or computer-readable media that are not only propagating transitory signals per se. In an aspect, tangible media can include non-transitory media wherein the term “non-transitory” herein as may be applied to storage, memory or computer-readable media, is to be understood to exclude only propagating transitory signals per se as a modifier and does not relinquish coverage of all standard storage, memory or computer-readable media that are not only propagating transitory signals per se. For the avoidance of doubt, the term “computer-readable storage device” is used and defined herein to exclude transitory media. Computer-readable storage media can be accessed by one or more local or remote computing devices, e.g., via access requests, queries or other data retrieval protocols, for a variety of operations with respect to the information stored by the medium.

Communications media typically embody computer-readable instructions, data structures, program modules or other structured or unstructured data in a data signal such as a modulated data signal, e.g., a carrier wave or other transport mechanism, and includes any information delivery or transport media. The term “modulated data signal” or signals refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in one or more signals. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

It can be noted that FIG. 11 describes software that acts as an intermediary between users and computer resources described in suitable operating environment 1100. Such software includes an operating system 1128. Operating system 1128, which can be stored on disk storage 1124, acts to control and allocate resources of computer system 1112. System applications 1130 take advantage of the management of resources by operating system 1128 through program modules 1132 and program data 1134 stored either in system memory 1116 or on disk storage 1124. It is to be noted that the disclosed subject matter can be implemented with various operating systems or combinations of operating systems.

A user can enter commands or information into computer 1112 through input device(s) 1136. As an example, a mobile device and/or portable device can include a user interface embodied in a touch sensitive display panel allowing a user to interact with computer 1112. Input devices 1136 include, but are not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, cell phone, smartphone, tablet computer, etc. These and other input devices connect to processing unit 1114 through system bus 1118 by way of interface port(s) 1138. Interface port(s) 1138 include, for example, a serial port, a parallel port, a game port, a universal serial bus (USB), an infrared port, a Bluetooth port, an IP port, or a logical port associated with a wireless service, etc. Output device(s) 1140 and a move use some of the same type of ports as input device(s) 1136.

Thus, for example, a USB port can be used to provide input to computer 1112 and to output information from computer 1112 to an output device 1140. Output adapter 1142 is provided to illustrate that there are some output devices 1140 like monitors, speakers, and printers, among other output devices 1140, which use special adapters. Output adapters 1142 include, by way of illustration and not limitation, video and sound cards that provide means of connection between output device 1140 and system bus 1118. It should be noted that other devices and/or systems of devices provide both input and output capabilities such as remote computer(s) 1144.

Computer 1112 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1144. Remote computer(s) 1144 can be a personal computer, a server, a router, a network PC, cloud storage, cloud service, a workstation, a microprocessor based appliance, a peer device, or other common network node and the like, and typically includes many or all of the elements described relative to computer 1112.

For purposes of brevity, only a memory storage device 1146 is illustrated with remote computer(s) 1144. Remote computer(s) 1144 is logically connected to computer 1112 through a network interface 1148 and then physically connected by way of communication connection 1150. Network interface 1148 encompasses wire and/or wireless communication networks such as local-area networks (LAN) and wide-area networks (WAN). LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring and the like. WAN technologies include, but are not limited to, point-to-point links, circuit-switching networks like Integrated Services Digital Networks (ISDN) and variations thereon, packet switching networks, and Digital Subscriber Lines (DSL). As noted below, wireless technologies may be used in addition to or in place of the foregoing.

Communication connection(s) 1150 refer(s) to hardware/software employed to connect network interface 1148 to bus 1118. While communication connection 1150 is shown for illustrative clarity inside computer 1112, it can also be external to computer 1112. The hardware/software for connection to network interface 1148 can include, for example, internal and external technologies such as modems, including regular telephone grade modems, cable modems and DSL modems, ISDN adapters, and Ethernet cards.

The above description of illustrated embodiments of the subject disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosed embodiments to the precise forms disclosed. While specific embodiments and examples are described herein for illustrative purposes, various modifications are possible that are considered within the scope of such embodiments and examples, as those skilled in the relevant art can recognize.

In this regard, while the disclosed subject matter has been described in connection with various embodiments and corresponding Figures, where applicable, it is to be understood that other similar embodiments can be used or modifications and additions can be made to the described embodiments for performing the same, similar, alternative, or substitute function of the disclosed subject matter without deviating therefrom. Therefore, the disclosed subject matter should not be limited to any single embodiment described herein, but rather should be construed in breadth and scope in accordance with the appended claims below.

As it employed in the subject specification, the term “processor” can refer to substantially any computing processing unit or device comprising, but not limited to comprising, single-core processors; single-processors with software multithread execution capability; multi-core processors; multi-core processors with software multithread execution capability; multi-core processors with hardware multithread technology; parallel platforms; and parallel platforms with distributed shared memory. Additionally, a processor can refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Processors can exploit nano-scale architectures such as, but not limited to, molecular and quantum-dot based transistors, switches and gates, in order to optimize space usage or enhance performance of user equipment. A processor may also be implemented as a combination of computing processing units.

In the subject specification, terms such as “store,” “storage,” “data store,” data storage,” “database,” and substantially any other information storage component relevant to operation and functionality of a component, refer to “memory components,” or entities embodied in a “memory” or components comprising the memory. It will be appreciated that the memory components described herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory.

As used in this application, the terms “component,” “system,” “platform,” “layer,” “selector,” “interface,” and the like are intended to refer to a computer-related entity or an entity related to an operational apparatus with one or more specific functionalities, wherein the entity can be either hardware, a combination of hardware and software, software, or software in execution. As an example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration and not limitation, both an application running on a server and the server can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media, device readable storage devices, or machine readable media having various data structures stored thereon. The components may communicate via local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or across a network such as the Internet with other systems via the signal). As another example, a component can be an apparatus with specific functionality provided by mechanical parts operated by electric or electronic circuitry, which is operated by a software or firmware application executed by a processor, wherein the processor can be internal or external to the apparatus and executes at least a part of the software or firmware application. As yet another example, a component can be an apparatus that provides specific functionality through electronic components without mechanical parts, the electronic components can include a processor therein to execute software or firmware that confers at least in part the functionality of the electronic components.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. Moreover, articles “a” and “an” as used in the subject specification and annexed drawings should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.

Moreover, terms like “user equipment (UE),” “mobile station,” “mobile,” subscriber station,” “subscriber equipment,” “access terminal,” “terminal,” “handset,” and similar terminology, refer to a wireless device utilized by a subscriber or user of a wireless communication service to receive or convey data, control, voice, video, sound, gaming, or substantially any data-stream or signaling-stream. The foregoing terms are utilized interchangeably in the subject specification and related drawings. Likewise, the terms “access point (AP),” “base station,” “NodeB,” “evolved Node B (eNodeB),” “home Node B (HNB),” “home access point (HAP),” “cell device,” “sector,” “cell,” and the like, are utilized interchangeably in the subject application, and refer to a wireless network component or appliance that serves and receives data, control, voice, video, sound, gaming, or substantially any data-stream or signaling-stream to and from a set of subscriber stations or provider enabled devices. Data and signaling streams can include packetized or frame-based flows.

Additionally, the terms “core-network”, “core”, “core carrier network”, “carrier-side”, or similar terms can refer to components of a telecommunications network that typically provides some or all of aggregation, authentication, call control and switching, charging, service invocation, or gateways. Aggregation can refer to the highest level of aggregation in a service provider network wherein the next level in the hierarchy under the core nodes is the distribution networks and then the edge networks. UEs do not normally connect directly to the core networks of a large service provider but can be routed to the core by way of a switch or radio area network. Authentication can refer to determinations regarding whether the user requesting a service from the telecom network is authorized to do so within this network or not. Call control and switching can refer determinations related to the future course of a call stream across carrier equipment based on the call signal processing. Charging can be related to the collation and processing of charging data generated by various network nodes. Two common types of charging mechanisms found in present day networks can be prepaid charging and postpaid charging. Service invocation can occur based on some explicit action (e.g. call transfer) or implicitly (e.g., call waiting). It is to be noted that service “execution” may or may not be a core network functionality as third-party network/nodes may take part in actual service execution. A gateway can be present in the core network to access other networks. Gateway functionality can be dependent on the type of the interface with another network.

Furthermore, the terms “user,” “subscriber,” “customer,” “consumer,” “prosumer,” “agent,” and the like are employed interchangeably throughout the subject specification, unless context warrants particular distinction(s) among the terms. It should be appreciated that such terms can refer to human entities or automated components (e.g., supported through artificial intelligence, as through a capacity to make inferences based on complex mathematical formalisms), that can provide simulated vision, sound recognition and so forth.

Aspects, features, or advantages of the subject matter can be exploited in substantially any, or any, wired, broadcast, wireless telecommunication, radio technology or network, or combinations thereof. Non-limiting examples of such technologies or networks include Geocast technology; broadcast technologies (e.g., sub-Hz, ELF, VLF, LF, MF, HF, VHF, UHF, SHF, THz broadcasts, etc.); Ethernet; X.25; powerline-type networking (e.g., PowerLine AV Ethernet, etc.); femto-cell technology; Wi-Fi; Worldwide Interoperability for Microwave Access (WiMAX); Enhanced General Packet Radio Service (Enhanced GPRS); Third Generation Partnership Project (3GPP or 3G) Long Term Evolution (LTE); 3GPP Universal Mobile Telecommunications System (UMTS) or 3GPP UMTS; Third Generation Partnership Project 2 (3GPP2) Ultra Mobile Broadband (UMB); High Speed Packet Access (HSPA); High Speed Downlink Packet Access (HSDPA); High Speed Uplink Packet Access (HSUPA); GSM Enhanced Data Rates for GSM Evolution (EDGE) Radio Access Network (RAN) or GERAN; UMTS Terrestrial Radio Access Network (UTRAN); or LTE Advanced.

What has been described above includes examples of systems and methods illustrative of the disclosed subject matter. It is, of course, not possible to describe every combination of components or methods herein. One of ordinary skill in the art may recognize that many further combinations and permutations of the disclosure are possible. Furthermore, to the extent that the terms “includes,” “has,” “possesses,” and the like are used in the detailed description, claims, appendices and drawings such terms are intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.

While the various embodiments are susceptible to various modifications and alternative constructions, certain illustrated implementations thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the various embodiments to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the various embodiments.

In addition to the various implementations described herein, it is to be understood that other similar implementations can be used or modifications and additions can be made to the described implementation(s) for performing the same or equivalent function of the corresponding implementation(s) without deviating therefrom. Still further, multiple processing chips or multiple devices can share the performance of one or more functions described herein, and similarly, storage can be effected across a plurality of devices. Accordingly, the various embodiments are not to be limited to any single implementation, but rather are to be construed in breadth, spirit, and scope in accordance with the appended claims. 

What is claimed is:
 1. A method, comprising: based on information comprising a topology of a network within an autonomous system, selecting, by a virtual route reflector of a device comprising a processor, a lowest cost path from paths emanating from a selected internal node of internal nodes of the autonomous system and exiting the autonomous system at respective border routers at a border of the autonomous system that are able reach a destination node external to the autonomous system via an external network that is external to the network; and transmitting, by the virtual route reflector to a client router in the autonomous system, a route from the selected internal node to the destination node, the route comprising a border router of the respective border routers at which the lowest cost path exits the autonomous system.
 2. The method of claim 1, wherein the lowest cost path is determined to be the border router having a nearest point of egress from the autonomous system relative to the selected internal node.
 3. The method of claim 1, wherein the lowest cost path is determined to be the border router having a lowest overhead to send a packet between the selected internal node and the border router.
 4. The method of claim 1, wherein the information further comprises external route information that identifies destination nodes that are able to be reached by the respective border routers of the autonomous system.
 5. The method of claim 1, further comprising performing, by the virtual route reflector, the selecting and the transmitting for all of the internal nodes of the autonomous system.
 6. The method of claim 1, further comprising performing, by the virtual route reflector, the selecting and the transmitting for the selected internal node to another destination node external to the autonomous system via the external network.
 7. The method of claim 1, wherein the topology comprises respective costs for respective links of the topology.
 8. A server, comprising: a processor; and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising: selecting, using information comprising a topology of an internal network within an autonomous system, a lowest cost path from paths emanating from a selected node of nodes within the autonomous system and exiting the autonomous system at respective edge routers, at an edge of the autonomous system, that are able reach an external node outside of the autonomous system via an external network; and transmitting, to a client router within the autonomous system, a route from the selected node to the external node, the route comprising an edge router of the respective edge routers at which the lowest cost path exits the autonomous system.
 9. The server of claim 8, wherein the lowest cost path is determined to be the edge router having a nearest exit point from the autonomous system relative to the selected node.
 10. The server of claim 8, wherein the lowest cost path is determined to be the edge router having a lowest overhead to send a packet between the selected node and the edge router.
 11. The server of claim 8, wherein the information further comprises external route information that identifies external nodes to which edge routers of the autonomous system are able to connect.
 12. The server of claim 8, wherein the operations further comprise iteratively performing the selecting and the transmitting for the nodes of the autonomous system.
 13. The server of claim 8, wherein the operations further comprise performing the selecting and the transmitting for the selected node to another external node outside of the autonomous system via the external network.
 14. The server of claim 8, wherein the topology comprises respective costs for respective links specified by the topology.
 15. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a device, facilitate performance of operations, comprising: selecting, using information comprising a topology of a network within an autonomous system, a lowest cost path from paths emanating from a selected node of nodes within the autonomous system and exiting the autonomous system at respective boundary routers at a boundary of the autonomous system that are able reach an external destination outside of the autonomous system via an external network; and transmitting, to a client router within the autonomous system, a route from the selected node to the external destination, the route comprising a boundary router of the respective boundary routers at which the lowest cost path exits the autonomous system.
 16. The non-transitory machine-readable medium of claim 15, wherein the lowest cost path is determined to be the boundary router having a nearest exit point from the autonomous system relative to the selected node.
 17. The non-transitory machine-readable medium of claim 15, wherein the lowest cost path is determined to be the boundary router determined to have a lowest overhead to send a packet between the selected node and the boundary router.
 18. The non-transitory machine-readable medium of claim 15, wherein the information further comprises external route information that identifies external destinations that boundary routers of the autonomous system have been determined to be able to reach.
 19. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise iteratively performing the selecting and the transmitting for a defined group of the nodes of the autonomous system.
 20. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise performing the selecting and the transmitting for the selected node to another external destination outside of the autonomous system via the external network. 